RAG Retrieval Strategies: Match 7 Methods to Failure Types
Diagnose missing evidence, poor ranking, wrong scope, stale values, and lost context before selecting a RAG retrieval method.
Choose a retrieval method based on what the search misses: exact terms, linked facts, the right scope, current values, or surrounding context. If the evidence is already present but ranked too low, use reranking.
Key takeaways
Match each retrieval problem to a method:
- Check the failure stage first: Separate missing evidence from poor ordering, lost context, and stale source data.
- Add hybrid search for queries that combine concepts with exact names, codes, or phrases.
- Use relations or metadata when a question depends on linked facts or a known scope.
- Rerank candidates when the first search finds relevant passages but places them too low.
- Query structured data or iterate when the task needs a current value or another retrieval pass.
How do you tell which part of retrieval is failing?
Start with a small set of representative questions and record the expected passages, values, or linked records.
A RAG evaluation guide from NVIDIA separates retrieval checks from answer checks. Recall asks whether reference evidence was retrieved; precision asks how much retrieved context is relevant (some metrics also reflect rank); faithfulness asks whether the answer is supported by that context. The ARES framework likewise treats context relevance, answer faithfulness, and answer relevance as distinct dimensions.
Diagnose a failure by checking whether the source contains the evidence, whether relevant results are ranked and scoped correctly, whether passages retain needed context, and whether the answer uses that evidence accurately. Evaluate representative queries and confirm improvements on held-out questions.
1. When should you combine keyword and semantic search?
Choose hybrid search when a question combines meaning with an exact name, code, ID, or phrase. Lexical search helps find exact terms, while semantic search can find related meaning despite different wording.
Lexical search is not always literal string matching: analyzed text may be tokenized or normalized, and exact matching depends on field configuration and query type. See OpenSearch text analysis for how analyzers affect matching.
OpenSearch's RRF documentation describes reciprocal rank fusion (RRF), which combines BM25 and vector result ranks rather than comparing scores on different scales. OpenSearch recommends it as a starting point when relative keyword and semantic weights are undecided.
For teams working with shared organizational evidence, AKB combines hybrid semantic-and-keyword retrieval with explicit relations between knowledge items, helping connect evidence across shared notes and linked records.
2. When does graph retrieval help with multi-hop questions?
Choose graph or relationship retrieval when a question depends on following links between entities or records; it can surface a connected path that separate similarity matches may miss. For example, “Which service owns the component that called this deprecated API?” may require relations across service, component, and API records.
Similarity search can find a passage about each term, while graph traversal can follow explicit connections between them. Neo4j's GraphRAG guide combines an ingestion process that extracts entities and relationships from unstructured text with graph algorithms for enrichment and summarization. Include the work of creating, checking, and updating those relationships in the design.
3. What can reranking fix when the first search pass is noisy?
Choose reranking when the first search finds relevant passages but places them below less useful ones. A reranker reorders the candidates; it cannot recover evidence the first search missed, so low recall requires a broader or different first-stage search.
Cohere's reranking guide describes reranking as a second stage after lexical or semantic retrieval.
4. When should you use metadata filters?
Choose metadata filters when the question defines a scope, such as a project, document type, release, department, or source. Filters limit eligible records; similarity scoring then estimates how closely they match the query.
Pinecone's metadata-filter documentation notes that filters are applied before scoring. Incorrect or missing metadata can exclude the record the question needs.
Filtering suits stable, queryable attributes. If the user asks for “the latest approved policy,” the system needs both a reliable status or version field and a rule for which values count as latest or approved.
5. When is a direct structured lookup better?
Choose a structured lookup when the user needs an exact field, count, status, or value maintained in a database or API. Examples include an account's current plan, inventory count, deployment state, or the owner ID stored in a service registry.
A parameterized query or API call can retrieve a changing value more directly than searching explanatory text. PostgreSQL's SELECT documentation covers selecting from tables or views and filtering rows with WHERE. Define what the agent may return according to the source system's authorization rules.
Use document retrieval when the user needs explanation, policy language, or context around the field. Use a structured lookup when the user needs the field itself. Some workflows need both: fetch the current value from the system of record, then retrieve the policy that explains what it means.
6. How does hierarchical retrieval preserve long-document context?
Choose hierarchical retrieval when a matching passage needs its surrounding section to make sense. The LlamaIndex hierarchical parser creates recursive chunk levels and links child chunks to parents. Compare a flat match with a child expanded to its parent, using queries that depend on a nearby definition, exception, heading, or table label. Retain enough context to preserve meaning without adding unnecessary text or cost.
7. When should an agent rewrite, decompose, or repeat a query?
Choose iterative retrieval when the first search leaves a specific evidence gap, the question has several parts, or a result reveals a useful term for another search. In LangGraph's agentic RAG guide, a relevance grader routes useful results to answer generation and irrelevant ones to a rewritten query and another retrieval pass.
Decompose questions whose parts need separate sources, rewrite wording that limits retrieval, and repeat a search to address an identified gap. Stop when evidence is sufficient or a defined retry or time budget is reached; report any unresolved gap rather than looping.
Which method matches the failure?
| Observed failure | Method to test first | What to measure |
|---|---|---|
| Exact identifiers or names are missed while related passages appear | Test hybrid keyword and semantic retrieval | Recall of exact-term questions and relevance of concept questions |
| An answer needs a chain of linked entities or records | Graph or relationship retrieval | Whether the required links are present and the answer path is correct |
| Useful candidates appear below noisy results | Reranking | Relevance of the top results and added latency |
| Results cross a known project, version, or document boundary | Metadata filters | Scope correctness and retrieval coverage within the filter |
| A user requests a current field, count, or status | Structured lookup | Value accuracy, source freshness, and access behavior |
| A small matching chunk may need surrounding material | Test hierarchical parent-child retrieval | Whether linked levels retain needed context at an acceptable context cost |
| The first pass leaves a specific evidence gap | Query rewrite, decomposition, or conditional retry | Gap closure, retry count, latency, and final answer quality |
FAQ
What is the best retrieval strategy for RAG?
Choose a strategy by the failure you measured, then test it on representative questions. Try hybrid retrieval for exact terms alongside semantic queries, graph or relationship retrieval for linked-record questions, reranking for noisy candidate order, metadata filters for known scope, structured lookup for exact fields, hierarchical chunking when parent context may help, and conditional rewriting when retrieved documents are judged irrelevant.
How does RAG do retrieval?
RAG searches an external knowledge source for evidence related to a user's question, then supplies retrieved context to a language model to help form an answer. Systems can use one retrieval method or combine methods, such as lexical and semantic search, followed by filtering or reranking, depending on the task.
Read the AKB documentation
Explore AKB's hybrid retrieval and explicit knowledge relations for shared organizational evidence.
