October 12, 2026

Building an Acceptance Test for Agent Knowledge Retrieval

How to design a grounded answer test set, isolate retrieval accuracy evaluation from generation noise, and run a fair vendor bake-off on internal documents.

An acceptance test should show whether a knowledge system can answer the questions your agents will actually face. Start with a small, representative set of questions and expected evidence, then evaluate retrieval and answer quality separately.

For engineering and knowledge-platform teams, the test is a pre-release decision gate: determine whether a candidate workflow is safe and useful for its intended users, sources, and permissions. Treat the sequence below as a repeatable evaluation plan, not a universal scorecard.

1. Choose the workflow and decision

Select a specific agent workflow, user identity, source corpus, and expected action. Define the release decision before testing: which failures block release, who owns remediation, who approves thresholds, and who reruns the test. Set thresholds with the stakeholders before seeing results; do not infer a pass rate or service-level target from a sample test set.

2. Build a versioned, permission-aware test set

Sample representative sources and query types from the intended workflow. For each case, record the source URI and revision (or equivalent immutable identifier), test identity and permissions, expected answer or abstention, required evidence, retrieval unit, and reviewer. Keep tuning examples separate from held-out regression cases. Record the corpus snapshot, query set, test identities and permissions, workload, and evaluation budget; hold these comparison conditions constant across candidates. Record each candidate's index, chunking, and retrieval configuration, varying those intentionally as the configurations under test. Record model and prompt versions as applicable.

The following is a proposed test-case template, not measured results. Replace the placeholders with approved sources, identities, expected behavior, and reviewer decisions before running the test.

3. Evaluate retrieval on its own

Run the query against the fixed corpus and inspect the ranked results before generation. Define the evaluation unit consistently as a document or a section/chunk; do not mix units within a comparison. For each query, label the relevant items and the separate set of required evidence items.

• MRR (Mean Reciprocal Rank) is the mean of the reciprocal rank of the first relevant result across queries. If the first relevant result is at rank 3, that query's reciprocal rank is 1/3.
• Precision@K is the number of labeled relevant items in the top K divided by K. Recall@K is the number of labeled relevant items in the top K divided by the total number of labeled relevant items for that query, using the same document or section unit for both counts.
• MAP and NDCG can summarize ranked relevance, but do not by themselves guarantee that every required piece of evidence for a multi-hop answer was found. Separately measure required-evidence coverage as the number of required evidence items retrieved in the top K divided by the total number of required evidence items for that query.
• Use no-answer and denied-access cases as separate outcomes; do not score them as ordinary relevant-result queries. Query complexity does not dictate one metric: select measures that match the expected evidence and release risks.

High context relevance means retrieved material appears pertinent; it does not prove that retrieval found all required evidence. Record missing evidence separately from irrelevant or noisy results.

4. Evaluate the generated answer

Only after retrieval inspection, run the answer-generation stage and map each factual claim to its supporting retrieved evidence. Track two distinct failure types: unsupported generation, where the answer adds or distorts a claim not supported by retrieved context; and missing evidence, where the retrieval results omit a required source or claim. An answer that happens to be correct from model memory does not repair a retrieval failure. Conversely, complete retrieved evidence does not excuse an unsupported answer.

5. Apply operational and access gates

Test permission checks after revocation, source deletion, and changes to group membership. Verify index lag and cache behavior, then revalidate citations against the current source and the requesting identity. If current authorization cannot be verified, the test should require blocking the response; do not present this as a product guarantee. Do not claim that content already delivered to a user can be recalled.

Measure p95 latency with the sample size, workload, and concurrency recorded. Track query and ingestion costs separately under the same workload. Set data-handling rules for evaluation logs: do not require unconditional retention of sensitive raw queries; minimize or redact them and retain only approved diagnostic metadata where possible.

6. Check the deployed integration

For an MCP integration, record the deployed server, client, proxy, and protocol versions. Test tool discovery, authentication, expected authorization boundaries, error propagation, and behavior for the protocol version actually deployed. MCP session and negotiation behavior varies by protocol version, so do not treat all MCP deployments as stateless. Tree browsing, search behavior, and relation traversal are implementation capabilities to verify in the selected product and client, not guarantees of MCP itself.

For product selection, AKB is a knowledge store for versioned documents, tables, files, relations, and search. Its current README describes a PostgreSQL-native revision store and says explicit Bare Git remains supported for existing deployments; do not assume every installation is Git-backed. Document revision history does not by itself establish whole-system rollback for tables, files, or indexes, nor complete audit coverage. Treat collection and curation workflows as deployment-specific or optional, and verify the exact release and configuration. Check the applicable release's terms in its official LICENSE before adoption.

7. Triage failures and retest

For each failed case, assign the primary cause—source or revision setup, permissions, retrieval coverage/ranking, answer grounding, integration, or operations—along with an owner and retest date. Preserve the original held-out cases for regression; add new cases only through a reviewed process so tuning does not silently turn the acceptance set into training data. After a fix, rerun the held-out cases with the original corpus, queries, identities, permissions, workload, and budget held fixed; change only the intended fix and record any necessary configuration delta. Record the reviewer disposition and release decision.

Practical questions

What does a relevant result miss? Compare the returned evidence with the complete required-evidence set. Relevant but incomplete context is a coverage failure, not proof of retrieval success.

What should happen when access is uncertain? Block or abstain in the test case unless the requesting identity's current access can be verified. Separately test revocation, deletion, index lag, and cache behavior.

What makes a vendor comparison fair? Use the same corpus snapshot, test identities, queries, evaluation units, thresholds, and approved workload. Record each system's own index and configuration rather than silently changing the comparison conditions.

Related reading

Which Sources Belong in an Agent Knowledge Base First?

October 12, 2026

Where Human Approval Belongs in Automated Knowledge Cleanup

October 12, 2026

Tracing an Agent Answer Back to a Source Version

October 12, 2026