How to Evaluate an Agent Knowledge Layer
An evaluation guide for engineering teams selecting an AI agent knowledge layer: comparing hybrid BM25 and vector search, shared Model Context Protocol backends across Cursor and Claude Code, and Git-backed version history for audit rollback.
An agent knowledge layer should help people and agents find the right evidence, understand its scope, and use it under the correct permissions. Evaluate that behavior with a repeatable workload before comparing feature lists. Retrieval, memory, governance and integration overlap, but each solves a different part of the problem.
Key takeaways
- Compare options against the same required workflow, source corpus, identities and evidence—not a single headline accuracy score.
- Test exact identifiers, semantic questions, tables, relationships, changing facts and questions that should remain unanswered.
- Measure retrieval quality, answer support, freshness, security and operating cost separately.
- Shared storage and MCP integration need identity, compatibility and concurrency checks across human and agent interfaces.
Start with the workflow and ownership boundary
A search engine, employee search application, data catalog, agent memory service and knowledge layer are not interchangeable products. A search engine can supply retrieval components; a catalog can organize metadata; a client memory feature can retain context for a particular workflow. Any of these may participate in a broader solution.
Write down what must be stored, who owns the source, who may read or change it, and which interfaces need access. Decide whether the proposed layer replaces a repository or sits alongside it. For a sidecar deployment, identify the system of record, deletion policy, synchronization owner and conflict rule. Reuse an existing component when it meets the requirement, and count the integration work needed to close any gaps.
Build a small, reproducible evaluation set
As a hypothetical starting point, use 50 questions drawn from a bounded business workflow. This is a planning example, not an industry standard or sufficient proof of production readiness. Include exact product codes, paraphrases, long-document evidence, table cells, related records, effective-date questions and unsupported requests. Run permission and failure scenarios separately, including identities that must be denied access.
Label the relevant evidence and acceptable answers with a domain reviewer. Freeze the corpus revision, relevance judgments, retrieval configuration, filters, model, prompt, context budget and deployment resources. Separate tuning cases from held-out evaluation. Report failures by query type so a strong average does not conceal a weak table or permission path.
External research can guide coverage without substituting for this test. LongMemEval, published at ICLR 2025, contains 500 questions spanning five memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. It evaluates long-term conversational memory; it does not certify enterprise security or establish an AKB production performance figure.
Check what retrieval actually preserves
Dense retrieval can help with paraphrases, while lexical ranking can help with terms. Neither guarantees exact identifier matching. Test tokenization of punctuation and codes, and use structured or exact-match filters where the application requires equality. Hybrid fusion and graph traversal are techniques to evaluate, not guarantees of correct answers.
Chunking can separate a passage from its meaning. Check parent headings, table row and column alignment, units, footnotes, dates and source references. Uploading a file does not demonstrate that its tables or embedded content were parsed correctly. Retrieve enough related material to support the task without assuming every linked document belongs in the answer.
| Test | Evidence to record |
|---|---|
| Exact identifiers and semantic questions | Expected records, ranked results and reasons for misses |
| Tables and related documents | Correct cell, units, scope and supporting relationships |
| Contradictory or dated evidence | Applicable revision and an explanation of unresolved conflict |
| Unsupported question | Appropriate abstention or a clearly bounded answer |
| Updated or deleted source | Direct-read visibility and search visibility measured separately |
Keep quality metrics distinct
For a question with labeled relevant evidence, Recall@5 is the number of relevant items retrieved in the top five divided by all labeled relevant items. State the aggregation method and handle cases with no relevant evidence separately. Recall is not answer accuracy: a model may ignore the right passage or generate an unsupported conclusion.
Report answer correctness against the reviewed rubric, citation support against the cited source revision, and abstention on unanswerable cases. Define each denominator. For latency, state whether the measurement covers retrieval alone or the complete answer, report p50 and p95, and retain timeout and failure counts rather than silently excluding them.
Test shared access, permissions and concurrent changes
MCP can reduce custom integration work, but each client still needs compatible transport, configuration and identity handling. A stdio server is commonly a subprocess; several clients need a common backend if they are meant to share state. A proxy is one possible arrangement. Persistent client-native memory can also exist, so compare actual storage and sharing boundaries rather than assuming all client memory is temporary.
Use two agent clients and a human UI or API workflow to read and update the same authorized record. Confirm record identity, revision, conflict behavior and the distinction between a committed write and an updated search index. Shared storage does not make prompts, cached context or generated answers identical.
| Acceptance case | Required observation |
|---|---|
| Denied access | No unauthorized content through bodies, snippets, titles, counts, relationships, provenance, caches, exports or derived text within the tested paths |
| Permission revocation | Access stops within a defined propagation target, including existing sessions and cached results |
| Concurrent writes | Documented conflict handling prevents silent loss of accepted changes |
| Timeout and retry | Retries do not create unintended duplicate changes; partial failures remain visible |
| Export and restore | Record counts, identifiers, relationships, permissions and sampled answers match the declared recovery scope |
Track unauthorized disclosures divided by unauthorized attempts, with a target of zero disclosures in the test. Correctly denied requests are successful security outcomes. Passing a finite test is not a universal security guarantee.
Separate provenance, history and recovery
A version counter, retained document revision, evidence citation, audit log and backup serve different purposes. A useful citation identifies the source, revision and location supporting a claim. Permission checks must also cover provenance. Git-based history alone does not establish immutable compliance evidence or a complete backup.
Ask what can be restored after losing the search index, primary database, binary storage or configuration. A local content update, model change, index rebuild and source re-parsing are different operations; they do not all require complete re-embedding. Measure the work required for the selected deployment and retained data. Test recovery point and recovery time objectives rather than relying on an architecture diagram.
Use AKB as a concrete evaluation candidate
The AKB public repository describes documents, tables, files, explicit relationships, retrieval and document history, with agent access through MCP and human or application access through UI/API workflows. These capabilities provide a starting point for a shared knowledge workflow; verify the chosen release and deployment against the worksheet.
Separate Core capabilities from optional Collector or Gardener workflows, and assign ownership for ingestion, review and recovery. Document history should not be generalized into line-by-line history for every table or binary file. Explicit relationships do not require an enterprise-wide formal ontology, and should not be presented as an automatic reasoning engine.
Check the release-specific license. AKB uses BSL 1.1; production use under the Additional Use Grant is limited to fewer than 100 aggregated Named Seats as defined there. Deployments at or above that threshold, or third-party hosted, embedded or rebranded offerings regardless of seat count, require a separate commercial license. Do not estimate eligibility from general company headcount.
Turn results into an operating decision
Record implementation hours, recurring review effort, infrastructure, model usage, support and recovery work. Cost per unique successful task should include failed and retried processing in its cost numerator without counting retries as additional successful tasks. If there are no successes, the ratio is undefined. Use measured workload assumptions and a stated time window; do not turn a small pilot into an unsupported savings claim.
Frequently asked questions
Can the evaluation fit into one month?
A month can be a planning example: scope and labels first, configuration next, held-out and failure tests after that, then a recovery drill and review. Actual timing depends on source access, risk and integration complexity.
Should the layer replace existing repositories?
Only if ownership, migration and operational requirements justify replacement. A sidecar can preserve existing systems of record, but synchronization and deletion behavior still need accountable owners.
How should licensing be compared?
Use the terms for the actual release, deployment and users, together with a current quote where needed. Include support and operating costs; source availability alone does not establish unrestricted production use.
Are embeddings enough for exact identifiers?
No. Evaluate exact-match or structured lookup alongside lexical and semantic retrieval, especially for punctuation-sensitive identifiers.
Does shared context eliminate drift?
No. It can reduce fragmentation, but clients may use different revisions, cached data, prompts or permissions. Test those boundaries explicitly.
Explore a shared knowledge layer
Evaluate AKB with your own evidence, identities and operational requirements. Learn more about AKB.
