RAG Reranking: How to Decide Whether You Need a Reranker

A retrieval team considering a reranker needs to measure its additional value on the candidates the current pipeline already returns. Compare reranking off and on with the same authorized candidates and context budget, then weigh the quality change against latency and cost. A public no-reranker result is a scoped reference for that test.

Key takeaways

Which retrieval features are hard gates?

Make correctness and access conditions hard gates; treat ranking changes as candidates for measured improvement. A ranking stage can bring a useful passage higher in the returned list, while a release gate states what the system must do for every relevant request.

Write each hard gate as an observable result. “Respect permissions” becomes a set of test requests tied to identities and sources, including a request after a permission is revoked. “Use current policy” becomes a question whose answer changes between two dated policy versions, with the expected source version recorded. “Retrieve enough evidence” becomes a defined query set and a minimum score or pass rate that the team chooses before comparing configurations.

A high relevance score cannot make an inaccessible record eligible for a user. Teams should enforce the applicable identity and source rules in the path that controls which records can reach the answer context, and test denied requests as directly as successful ones.

Where does reranking fit in a retrieval pipeline?

Reranker limitations. A reranker is a second scoring stage that reorders candidates before model context is assembled. It cannot add a passage missing from the initial candidate list, so its possible benefit depends on both candidate recall and its ability to distinguish the candidates it receives.

If using an external reranker, verify that candidate text and metadata are authorized for that service and meet its retention and data-residency constraints. A user’s eligibility to access a record does not, by itself, authorize sending it to a third party.

Experimental controls. For a controlled ranking test, hold the first-stage retriever, eligibility rules, deduplication, candidate IDs, and final context budget constant. Compare reranking off and on against the same candidate set to isolate the ranking effect. A separate whole-pipeline deployment comparison can measure end-to-end impact when components or operating conditions differ, but label it separately rather than attributing the difference to reranking alone.

1. Apply the request context. Identify the user, agent, knowledge space, and any applicable source or metadata constraints. These inputs define which material may be considered for the request.
2. Retrieve an initial candidate set. Use the system’s selected retrieval method, such as keyword search, vector search, or a hybrid. Save the retrieved records and their initial scores so both experimental configurations start from the same material.
3. Deduplicate and prepare candidates. Remove repeated copies when duplication would consume result slots, then present comparable text and source metadata to the ranking stage. Preserve IDs so each returned item can be traced to its source.
4. Score and reorder when enabled. Give the reranking stage the query and the candidate passages, then use its order or scores to select which candidates continue.
5. Choose context and generate an answer. Apply the same final context limit in both test runs, then use the same answer model and prompt. This keeps a retrieval comparison from quietly becoming a prompt, model, or context-size comparison.

The acceptance test should verify the result: permitted records can contribute, and disallowed records do not reach the model.

Which workloads should you test?

Keep an evaluation record with query, principal, expected source IDs and versions, acceptable evidence, and required answer elements. LongMemEval (2024 preprint; ICLR 2025) offers memory-task categories; the release decision uses your own corpus and workload.

Build the test set from retrieval failures your own users would notice, then add ordinary queries so the comparison represents routine use as well as edge cases. A test made only of simple keyword matches can miss the cases where ranking is most likely to affect the selected context. A test made only of difficult edge cases can overstate how often those cases occur in daily traffic.

Include these query families:

• Exact names and identifiers: Test whether an incident ID, product code, policy number, API name, or project retrieves its specific record rather than a generally related one.
• Natural-language paraphrases: Use alternate wording, abbreviations, synonyms, and role-based descriptions to test retrieval when query terms differ from the source.
• Near matches: Contrast similar but inapplicable passages, such as instructions for different versions or departments, and test whether the query distinguishes the correct one.
• Version and time questions: Ask what applied before or after a change, or when a rule took effect, and include the expected date and source version.
• Multi-source questions: Require two or more passages for a conclusion, and score whether all needed evidence appears in the selected context.
• Incomplete or unsupported questions: Include questions without a reliable source answer and check that the system follows its answer policy instead of treating a plausible near match as support.
• Routine questions: Sample common request types and proportions, including short direct questions, longer conversational requests, and searches with an obvious first useful passage.

How do you compare reranking on and off fairly?

Run a controlled comparison in which reranking is the only changed retrieval component, then measure retrieval, answer support, latency, and cost. The RAGAS evaluation paper separates the ability to retrieve relevant, focused passages from the model’s ability to use them faithfully and from the quality of the generated answer. That separation gives the test a practical structure: first inspect what the retrieval stage returned, then assess what the answer model did with it.

Prepare the experiment

1. Freeze the evaluation inputs. Use the same corpus snapshot, source versions, access configuration, query records, filters, and relevance judgments for both runs. Record the document-processing and deduplication settings that affect which records become candidates.
2. Choose a baseline. Run your current retrieval setup with reranking off. Save candidate IDs, rank positions, scores if available, final context, output, and runtime for each query.
3. Choose one reranked configuration. Hold the first-stage retriever, candidate budget, answer model, prompt, and final context budget steady. Change only the reranking stage or its defined settings for this comparison.
4. Select settings on a tuning set. Use one part of the query collection to choose candidate depth and reranker settings. Then freeze the choice before using a separate held-out set for the release comparison.
5. Repeat under representative operating conditions. If the service is concurrent or the corpus is large, run enough requests to observe the runtime behavior your users will see. Keep the hardware, deployment region, concurrency, and caching policy constant between configurations.

Measure retrieval separately from answers

Compare retrieved candidates and final context with the expected evidence IDs at the cutoffs the team selected for its test. State the retrieval unit and denominator: Hit@K is whether any labeled relevant item appears in the first K results; Recall@K is the fraction of labeled relevant items retrieved at that cutoff. Keep those measures separate from answer correctness. Report unsupported-question and denied-access outcomes separately from positive relevance cases.

When assessors can label degrees of relevance, use a graded ranking measure such as normalized discounted cumulative gain (NDCG) at a stated K. The Stanford information-retrieval text’s evaluation chapter explains that NDCG supports non-binary relevance judgments and measures the top results at a chosen cutoff. A top position can then receive more credit for a highly useful passage than for a merely related passage.

Next, score the final context and answer as separate outcomes. For the context, check whether it contains all required evidence, whether it includes conflicting or stale material, and whether the selected context budget keeps the relevant passages. For the answer, check whether each required claim is supported by that context, whether the model uses the correct source version, and whether the answer follows the product’s policy for unsupported questions.

Record the tradeoff

Measure time for the complete query path and, where instrumentation allows, the time added by reranking itself. Report typical and tail latency, such as median and p95, for the same workload and operating conditions. Track errors and timeouts alongside successful responses, since a configuration that improves average ranking but misses a service target under load may not suit the release.

Record costs using the actual pricing and deployment inputs for the system under test. Separate per-query ranking charges, additional compute, and any infrastructure capacity the selected configuration needs.

Include costs incurred by failed requests and retries in the total-cost numerator. Avoid double-counting a per-request service fee and the same compute expense already included in a bundled charge.

Use the expected query volume and concurrency in your calculation, and state those assumptions beside the result. This gives the team a concrete operating estimate without treating a provider’s advertised rate or a small local test as a universal cost.

Present outcomes by query family as well as as a total. One compact report can show the baseline and reranked result for retrieval recall at the chosen cutoff, NDCG or another ranking metric, answer-support pass rate, median and p95 latency, and estimated cost per query at the stated scale.

Report paired per-query differences, the number of paired queries, and an appropriate uncertainty estimate; a small observed gain is not automatically robust. Add a short list of corrected and regressed cases. The report should show whether the improvement serves the workload’s important failures and whether any gains arrive within the stated operating limits.

What does AKB’s documented no-reranker result show?

In its public repository, Dnotitia reports, in its vendor-run repository evaluation reviewed October 14, 2026, Recall@5 of 98.4% on LongMemEval-S, a 500-question dataset, for dense-plus-BM25 hybrid retrieval with source-level deduplication, bge-m3 embeddings at 1024 dimensions, and no reranker.

The benchmark gives a memory-oriented reference: the original LongMemEval paper describes LongMemEval-S histories of about 115,000 tokens per problem and questions that test long-term memory across dialogues, including temporal reasoning and knowledge updates. An individual developer can use AKB’s reported configuration as a starting reference for evaluating agent memory across long records; an enterprise team can test the same retrieval setup against its policies, technical sources, versions, and user groups.

The repository describes its comparison as stack-level because the compared systems use different embedding models. For a reranker decision, the important detail is that this reported AKB configuration uses no reranker; teams can compare it with a reranked configuration on their own fixed query set without assuming the benchmark settles that question. This retrieval result measures neither enterprise answer accuracy nor permission enforcement. Pin the evaluation code and data with the release under test, and record hardware, concurrency, and timing separately when measuring deployment latency.

When should reranking become a hard requirement?

Require reranking when the held-out test demonstrates a necessary improvement in a release gate and the selected setup meets the approved runtime and cost limits. Keep it as an optional optimization when the no-reranker configuration meets the same gates on representative queries and reranking adds no decision-relevant benefit.

Use this decision sequence:

1. Define the failure and threshold. Name the user-visible failure the reranker should address and set the metric, cutoff, acceptable result, and latency or cost limits before testing.
2. Check held-out results. Compare the baseline and frozen reranked setup on held-out queries; verify that it improves the named failure, inspect regressions by query family and source characteristics, and check that costs and latency remain within limits.
3. Record the configuration and evidence. Save the corpus revision, query set, model and index settings, authorization rules, candidate depth, context limit, relevance judgments, operating assumptions, and separate results for ranking, evidence support, task correctness, latency, and cost.

Related reading

Which Sources Belong in an Agent Knowledge Base First?

October 12, 2026

Where Human Approval Belongs in Automated Knowledge Cleanup

October 12, 2026

Tracing an Agent Answer Back to a Source Version

October 12, 2026