October 12, 2026

Choosing a Retrieval Mix for Agent Knowledge

How to balance exact keyword matching, dense semantic search, and document relation traversal for agent knowledge layers without over-engineering graph infrastructure.

Start with representative questions from the workflow you want an agent to support. Compare keyword, semantic and hybrid retrieval on that same set, then test whether explicit links help with questions that require following a reference. Choose the simplest mix that meets the observed requirements; no retrieval method is a universal winner.

Build a small, representative query set

Include exact identifiers, natural-language questions, mixed queries and questions that require following a known reference. Label the source passages that a useful result should contain before comparing configurations.

These are example test cases, not benchmark results. Use your own terminology and expected sources. Keep denied-access and no-answer cases in the test set as well.

Hybrid retrieval mechanics and reciprocal rank fusion

Parallel dense retrieval, BM25 keyword matching, and Reciprocal Rank Fusion (RRF) represent one common hybrid implementation, though hybrid search also encompasses score fusion, multi-stage pipelines, and dynamic query routing. Describing parallel retrieval branches is illustrative and does not claim that two candidate branches always execute physically in parallel in every execution runtime.

How reciprocal rank fusion operates

Dense vector search and lexical search produce raw scores on differing scales and distributions. Reciprocal Rank Fusion (RRF) bypasses score normalization by evaluating the ordinal rank positions of documents across candidate runs.

For each document d, RRF computes a composite score:

score(d) = sum over lists containing d of 1 / (c + rank_i(d))

Where rank_i(d) starts at 1 for the top item in retrieval list i, missing candidates in list i contribute zero, and c is the fusion constant (avoiding confusion with top-K candidate bounds). Implementation settings—such as Elasticsearch's default fusion constant c = 60—are engine-specific defaults rather than universal laws.

Because RRF evaluates inverse ranks across candidate sets, documents appearing near the top of multiple branches receive higher composite scores. Single-list candidates enter the merged candidate union, but may be excluded if final result truncation limits total output. RRF deliberately produces a new combined ranking order and cannot recover relevant documents missing from every candidate branch list.

Hybrid RRF vs. dynamic query routing

In dynamic query routing, an incoming query is evaluated by a classifier or regex filter to select keyword or semantic retrieval upfront.

Routing avoids executing redundant search branches, but a misclassified query can omit critical documents entirely. Parallel retrieval queries both engines and performs rank fusion. Evaluate candidate limits, missing items, and latency overhead on the target deployment rather than assuming one paradigm strictly dominates.

Evaluating retrieval configurations and explicit reference traversal

To select an effective retrieval mix, evaluate configurations across representative workloads under uniform test parameters: matching corpus versions, identical tenant and security filters, and fixed result or token budgets.

Configuration comparison framework

Compare four baseline configurations against the same query suite:

1. Lexical Retrieval (BM25): Evaluates term-matching baseline performance.
2. Dense Semantic Retrieval: Evaluates vector embedding semantic similarity.
3. Parallel Hybrid (Dense + Lexical + RRF): Evaluates candidate rank fusion.
4. Hybrid + Explicit Reference Traversal: Evaluates hybrid retrieval supplemented by following explicit URI links or structural citations for multi-hop queries.

For every evaluation run, record per-branch candidate limits (top-K), deduplication logic, fusion settings (such as constant c), embedding model identifier, reranker model (if used), and both cold/warm latency and concurrency metrics. Separate retrieval evaluation (whether the target context was retrieved) from generated answer quality (LLM synthesis correctness). Keep hyperparameter tuning datasets strictly separate from held-out evaluation sets.

Defining precise evaluation metrics

Avoid vague notions of "retrieval coverage." Measure concrete retrieval performance using defined parameters:

• Retrieval Unit: Define whether items are judged at document level or section/chunk level.
• Hit@K vs. Recall@K: Hit@K measures whether at least one labeled relevant passage appears in the top K results; Recall@K measures the fraction of total relevant passages retrieved in the top K.
• First Relevant Rank: Position r of the first labeled relevant result. Reciprocal rank is 1/r (0 if none within the evaluated cutoff); MRR is its mean over the specified query set.
• Latency & Operating Cost: Evaluated per query class (exact identifier, semantic paraphrase, multi-hop reference).

Cases involving unanswerable questions (no-answer queries) or denied access permissions must be tested and reported separately; do not fold negative or access-denied queries into standard relevance averages. Set workload-specific acceptance thresholds prior to testing.

Cross-document traversal: explicit links vs. conceptual navigation

Enterprise documentation rarely exists in self-contained, isolated paragraphs. Policies reference appendices, API reference docs cite configuration schemas, and architecture records link to technical dependencies.

Concept-based traversal vs. explicit reference navigation

Similarity search ranks passages based on term or vector proximity. It does not follow explicit relational structures intended by authors, nor does semantic proximity guarantee correct multi-hop logical reasoning.

Test reference navigation as a distinct capability: retrieve the starting record, inspect its stored URI or citation link, navigate to the target, and verify revision consistency.

Stored explicit relations vs. formal ontology reasoning

To navigate cross-document references, organizations evaluate structural storage patterns alongside semantic retrieval:

1. Formal Semantic Models: Formal semantic models may use RDF/OWL or other representations; reasoning requires an appropriate engine and defined semantics. A property graph or graph database alone does not imply formal ontology reasoning.
2. Explicit Stored Relations and Structural Links: Explicit document URIs, parent-child section hierarchies, and metadata links (depends_on, related_to) stored alongside versioned document content.

Explicit stored relations make known links directly navigable for agents, avoiding reliance on unassisted vector similarity jumps. However, explicit document links are not by themselves formal ontology reasoning engines, nor should all ontologies be categorized as inherently heavyweight or manual.

Implementing agent knowledge with Dnotitia AKB

Integrating a multi-strategy retrieval layer into agent environments benefits from standardized tool protocols, version-controlled storage, and explicit relational support.

Dnotitia develops AKB (Agent Knowledge Base), an AI data infrastructure component designed to supply organizational knowledge and memory to AI agents. The current public AKB architecture describes PostgreSQL Native revisions, with PostgreSQL as the source of truth for chunk text, metadata, and BM25 vocabulary. Explicit Bare Git remains supported as a legacy backend. File attachments use separate storage, while pluggable vector stores provide derived indexes for dense embeddings and corpus-side sparse vectors.

Key architectural capabilities

• Multi-Access Capabilities: Alongside hybrid keyword and semantic retrieval (akb_search), agents can selectively invoke direct URI lookup (akb_get), structural tree or section navigation (akb_browse, akb_drill_down), and literal grep string search (akb_grep) depending on task demands. Note that BM25 keyword matching depends on corpus tokenization and analyzer filters, so it is not guaranteed exact literal string matching for every string pattern.
• Explicit Relational Storage: Knowledge resources retain canonical URI identifiers and explicit relational metadata (depends_on, related_to), enabling direct graph traversal (akb_relations) alongside version-controlled document content.
• Standardized MCP Connectivity: Agents operating in environments such as Claude Code, Cursor, or custom execution runtimes query the knowledge layer using standardized MCP tools across permissioned vaults.

Robustness testing and deployment parameters

Before production deployment, execute targeted operational resilience tests:

• Embedding Provider Outage: Verify driver fallback behaviors during vector endpoint failures (for example, testing whether the vector driver degrades gracefully to BM25 lexical search). Fallback policies are driver- and configuration-specific rather than universal promises.
• Stale Vector Index & Broken Relations: Test retrieval performance when recent document edits have not yet finished indexing, or when target URIs in relational metadata are revoked, deleted, or point to legacy revisions.

Keep these proposed verification checks distinct from empirical benchmark claims; do not publish unverified performance numbers.

AKB is self-hostable and distributed under the BSL 1.1 License (with the akb-mcp client proxy licensed under MIT). Core deployments provide the read/write engine and retrieval interfaces, while automated ingestion pipelines operate as separate external components.

Related reading

Which Sources Belong in an Agent Knowledge Base First?

October 12, 2026

Where Human Approval Belongs in Automated Knowledge Cleanup

October 12, 2026

Tracing an Agent Answer Back to a Source Version

October 12, 2026