Proving a Knowledge Layer in One Real Workflow
How to choose, scope, and evaluate a single-workflow agent knowledge pilot: comparing role-specific onboarding against incident response triage within a 90-day window.
A knowledge-layer pilot should answer a bounded operational question: can an agent help complete one real workflow to an agreed standard, within its authority and at an acceptable operating cost? Compare the existing process with a controlled pilot on the same tasks. Treat results as evidence for that workflow, not proof of enterprise-wide benefit.
Key takeaways
- Define one workflow by trigger, users, source owners, decision, evidence, deliverable, and authority limit.
- Freeze the baseline and evaluation protocol; held-out cases reduce tuning leakage but do not guarantee unbiased results.
- Measure task success, evidence recall, claim support, both kinds of abstention error, failures, freshness, effort, and cost.
- Set expand, revise, and stop gates before results. Zero leaks in finite tests do not prove security.
- Test source changes, revocation, replay, restore, and approved writes in the actual deployment or an authorized sandbox.
Choose one bounded workflow
Onboarding and support or incident triage are possible candidates, not universal risk categories or established ROI rankings. Evaluate local task volume, consequences, evidence restrictions, and ability to score outcomes. This example uses Tier-2 product-support ticket triage:
- Trigger: A ticket is escalated to the Tier-2 queue.
- Users: Support engineers use their current interface; an agent retrieves approved knowledge through MCP.
- Owners: Product-documentation and support-runbook owners maintain sources; a security owner defines access.
- Decision: Classify the issue and identify applicable troubleshooting steps.
- Evidence: Ticket details, permitted diagnostics, current runbooks, and relevant product documentation.
- Deliverable: A draft response and next-step recommendation, with citations for factual claims.
- Authority limit: Retrieval and drafting only. A human reviews and sends the response. The agent cannot change production systems, grant itself access, or make generated facts authoritative.
This is an illustrative example, not a recommendation that every support team should choose triage. Keep task definition, users, evidence, permissions, and allowed actions constant between baseline and pilot. Record any protocol change instead of mixing unlike cases.
Freeze a comparable baseline
Before configuring the agent, document how the existing human workflow performs on the same task definition. Lock source revisions, permission sets, task cases, rubric, model and prompt versions, retrieval configuration, hardware, and cost-accounting rules. Record operator experience and task order. Repeat runs where feasible and account for learning, fatigue, and ordering effects.
Separate tuning cases from held-out evaluation cases, and keep held-out labels unavailable during tuning. This reduces tuning leakage; it does not guarantee unbiased results or eliminate all overfitting. Include ambiguous requests, stale sources, conflicting instructions, unanswerable tasks, and unauthorized requests. Evaluate read-only retrieval separately from authorized writes or production actions.
Pilot charter and acceptance worksheet
Agree on the rubric, denominator, thresholds, safety rules, and decision owners before evaluating results. Retrieval recall or precision measures evidence retrieval, not whether an answer is correct.
| Measure | Definition and denominator | Reporting rule |
|---|---|---|
| Task success | Tasks meeting the acceptance rubric / all eligible tasks attempted | Show numerator and denominator for baseline and pilot. |
| Evidence recall | Labeled relevant evidence retrieved / all labeled relevant evidence | Fix relevance labels before testing; this is not answer accuracy. |
| Supported checked claims | Supported factual claims checked / all factual claims checked | Define the claim-checking rubric and report both counts. |
| Abstention | Correct abstentions / unanswerable cases; separately, erroneous abstentions / answerable cases | Report both rates and denominators. |
| Unauthorized disclosure | Successful leaks / unauthorized test attempts | Report counts and test coverage. Finite zero-leak results are not proof of security. |
| Task duration | p50 and p95 for completed tasks | Report timeout and failure rates separately. |
| Freshness | Time from approved source change to expected searchable state | Set a deployment-specific target; report lag, pending updates, caches, and failures. |
| Human effort | Reviewer, source-maintenance, operations, and evaluation hours | Separate setup from recurring work. |
| Total cost per unique accepted task | Compute, infrastructure, and human effort—including failed attempts and retries—/ unique accepted tasks | Undefined if no tasks are accepted; do not divide by generated outputs. |
Use rubric review for correctness and set workflow-specific thresholds before results; there is no universal pass mark.
Hypothetical arithmetic—not a benchmark
Suppose, purely hypothetically, the existing process completes 18/30 = 60% of held-out tasks acceptably, and the pilot completes 24/30 = 80%. That is +20 percentage points and six additional acceptable tasks in this sample—not a 20% improvement. These figures are not actual results, a benchmark, or a forecast. A small paired sample does not establish population-level benefit, causality, or readiness for another workflow.
Stages and pre-set decision gates
A 90-day planning cap is one optional example, not an industry norm or promised deployment time. A team could scope and baseline, configure, test held-out cases, run a human-supervised trial, and review the evidence. Set dates to fit the real integration and review work.
Define gates before results are unblinded:
- Expand: Consider a limited next stage only if pre-agreed task, time, cost, freshness, and safety criteria are met and owners accept residual risks. Passing finite tests does not prove security.
- Revise: Fix a missed but remediable criterion, document the change, and retest on fresh held-out cases. Do not move thresholds after viewing results.
- Stop or pause: Stop expansion if the safety boundary is breached, a critical criterion remains unmet after retesting, or a serious unauthorized disclosure occurs. A serious leak pauses expansion regardless of average quality.
Operational tests and accountability
Run these checks against the actual deployment, connected stores, permissions, and caches:
- Source change and freshness: Edit an approved source and measure when it becomes searchable, including lag, pending work, and cache effects.
- Permission revocation: Revoke access and verify that subsequent unauthorized requests are denied. Do not assume revocation instantly refreshes every index.
- Replay and recovery: Test replay idempotency. Separately follow a documented backup and restore procedure across actual datastores; verify integrity, authorization, and rubric-based task behavior. A Git revision is not a universal backup, and restoration does not promise identical model answers.
- Approved sandbox write: In an isolated, authorized sandbox, test a conflict-safe approved update and its review and rollback path. Do not write into the live support or production workflow.
- Approved outcome reuse: Only after owner approval, add a resolution to the test knowledge set and assess its use on a later task. Generated content alone does not make a fact authoritative.
Name source/domain, security, and platform owners; assign human review for drafts and define a fallback to the existing support process.
AKB in brief
The AKB public repository, checked September 28, 2026, describes PostgreSQL-native revisions as the default and Bare Git support for legacy deployments. It provides a shared knowledge store with Web UI and REST access for people and applications, and MCP for agents. The Core store is distinct from external curation. Verify the installed version and configuration rather than assuming all deployments use the current default.
For the exact storage backend and access surfaces in a specific release, consult the current AKB repository and record what the pilot deployment actually uses. Do not infer propagation speed, cache behavior, permission outcomes, restore coverage, or answer determinism from a product description. Treat source approval and external curation as explicit pilot responsibilities, and test them in the selected environment.
Li et al., Knowledge Graphs in Practice (arXiv:2304.01311), interviewed 19 practitioners from enterprise and academic settings. The IEEE TVCG journal reference is January 2024; the initial preprint is from 2023. Findings suggest differing role needs, not that every system requires one interface design, and provide no ROI evidence for this pilot.
FAQ
What belongs in the first support-triage pilot?
One ticket type, a defined escalation trigger, named users and source owners, a fixed evidence set, a draft-only deliverable, and human approval before communication or action. Keep production changes outside agent authority.
How should baseline and held-out results be compared?
Freeze the task rubric, source revisions, permissions, prompts, model and runtime settings, and cost rules. Keep tuning and held-out cases separate. Holdout reduces tuning leakage but does not guarantee unbiased estimates; report outcomes with denominators.
Does evidence recall tell us whether an answer is correct?
No. Evidence recall measures retrieval of labeled relevant material. Separately score task success, supported checked claims, and correct and erroneous abstentions against fixed rubrics.
Does a source edit or permission revocation take effect immediately?
Do not assume so. Measure source freshness, pending work, caches, and permission behavior in the actual deployment; verify that later unauthorized requests are denied.
Does Git rollback restore all data or guarantee the same answer?
Do not assume either. Test replay idempotency and documented backup/restore across actual datastores, then check integrity, authorization, and rubric-based behavior—not exact model wording.
Is 90 days a standard pilot duration?
No. It is an optional planning example. Fix remediable issues and retest on fresh held-out cases without changing pre-set thresholds. A serious unauthorized disclosure pauses expansion even if average quality is high.
Learn more
Learn more about Dnotitia and evaluate AKB against your workflow requirements.