How to Check a Retrieval Benchmark for Model Disclosure and Reproducibility
Audit a retrieval benchmark by its metric, dataset size, embedding model, reranker, and public harness, using AKB's LongMemEval-S result as an example.
Five checks before comparing retrieval results
- Metric: In this runner, Recall@5 is the fraction of questions whose top five results include at least one annotated answer session. It is not answer accuracy or proof that all needed evidence was retrieved.
- Sample: The historical AKB result was 492 of 500 LongMemEval-S questions.
- Setup: That run used
bge-m3@1024, pgvector, and no reranker. - Reranking: A reranker can improve or worsen ranking; its effect depends on the system and test.
- Reproduction: The public methodology and category breakdown are useful, but the bundled Compose setup is archival and unsupported against the current backend.
A retrieval score means little until you know what produced it. Check five things before comparing numbers: what the metric counts, how many questions were tested, which embedding model and vector store were used, whether a reranker ran, and whether the result can be reproduced. The AKB repository reports a historical result of 98.4% Recall@5 on LongMemEval-S, measured on May 20, 2026 with backend b54184a.
1. What does the metric measure?
In this runner, Recall@5 is the fraction of questions for which at least one annotated answer_session_id appears among the top five results. It measures retrieval of an annotated answer session—not whether the system gives the right answer or retrieves every piece of evidence needed to answer.
AKB's reported 98.4% means that this condition was met for 492 of 500 questions in the historical run. Use answer-quality evaluation separately, and check how each benchmark defines its metric before comparing scores.
2. Which dataset, and how many questions?
LongMemEval tests long-term memory in chat assistants. The _s split used for the historical result has 500 questions, with about 50 chat sessions in each question's history to search.
The comparison below summarizes historical published pipeline results. It is not a current product ranking or a controlled comparison: systems used different embeddings, and the sample sizes and reranking setups differ.
3. Which embedding model and retrieval method?
The historical AKB run used bge-m3@1024 embeddings, hybrid dense and BM25 retrieval, source-level deduplication, and pgvector. The other systems use different embeddings, so score differences reflect whole-pipeline differences rather than an isolated comparison of retrieval methods.
For your own evaluation, record the backend version, model, vector store, and retrieval settings.
4. Was a reranker used?
The historical 98.4% AKB result used no reranker. In the same README, an AKB run with cross-encoder reranking and rank fusion scored 97.8% (489/500). Reranking can improve or worsen ranking, so test its effect on the same workload rather than assuming a gain.
A run without reranking omits that stage, but this score does not establish lower latency or cost than another product or setup. Measure latency and cost under the workload and configuration you plan to use.
5. Can you reproduce it?
The AKB methodology, runner, and per-category breakdown make the historical setup inspectable and offer useful guidance for designing an evaluation. The bundled Compose setup is archival and unsupported against the current backend; it is not a one-command way to reproduce the published score.
The repository does not pin an immutable backend image or complete historical revision, and the raw per-question results are not included. A current evaluation requires adapting and validating authentication, ingestion, indexing waits, and cleanup for the selected backend version. Use a disposable instance because the runner creates and deletes accounts and per-question vaults.
FAQ
Is Recall@5 the same as answer accuracy?
No. In this runner, Recall@5 counts questions where at least one annotated answer session appears in the top five results. It does not assess the generated answer or confirm that all needed evidence was retrieved.
Why are two systems at 98.4% not necessarily equivalent?
The historical figures used different pipelines and question counts; the AKB run used no reranker, while the MemPalace result used one. Treat them as published pipeline results, not a controlled comparison.
Evaluate on your own corpus
Review the methodology and plan an evaluation on your own corpus.
