How to Evaluate Long-context Retrieval
A large context window does not guarantee the model used the right passage. Read how to test position sensitivity, a worked example, and the cost.
A long context window lets a system hold more material, which makes it tempting to skip retrieval altogether. The decision has a hidden risk: a model can accept a long prompt and still fail to use the passage that matters, especially when it sits in the middle. The Lost in the Middle paper reports exactly that effect, which the long-context article covers for the practical consequence, as Reranking in Retrieval Pipelines: When and Why It Pays explains for why selection still matters.
Direct answer:
Evaluate long-context retrieval by testing whether the right passage is used across positions, with a fixed question set and a comparison against a shorter retrieved context.
The Lost in the Middle paper is the primary source, and the Sentence-BERT paper explains the bi-encoder retrieval that a shorter pipeline uses. The boundary: position behaviour is model-specific, which the benchmarks article warns about applying a result broadly, and the gain must be weighed against token cost, as the cost estimator article requires.

What to test
Position sensitivity. Place the relevant passage at the beginning, middle, and end of otherwise identical prompts and score the answer, as Long Context Windows: What a 1M-Token Window Actually Buys You recommends. A model that fails only in the middle has a position problem, which a better retriever can avoid by shortening the context, as Reranking in Retrieval Pipelines: When and Why It Pays explains.
Distractors. Real contexts contain similar-looking passages with conflicting details, which the hallucination article treats as a grounding problem.
Citation correctness. If the system names a source, check that the cited passage supports the claim, as the legal-research article requires for grounded answers, as the reduce-hallucinations article treats citation checks as a core control.
Cost per correct answer. A long prompt is billed on every request, which the prompt-caching article explains for repeated prefixes. Measure cost per correct answer rather than accuracy alone, as the cost estimator article advises.
Three consequences follow. The question set should be fixed, as AAPOR's standards require for comparable wording. The test should include a retrieved baseline, which the embedding-selection article recommends for model comparisons. And the result should be recorded with the cost, as the decision-memo article requires.
A worked example
The following is hypothetical and illustrative. A team considers sending an entire 200,000-token manual with each question instead of retrieving five passages.
Three checks decide. First, accuracy across positions on 100 questions, as Long Context Windows: What a 1M-Token Window Actually Buys You advises. If the model scores 92% when the passage is at the edges and 70% in the middle, the long context is unreliable. Second, cost: at the published input price, each full-manual request costs substantially more than a retrieved one, which the cost estimator calculates. Third, distractor behaviour: adding a conflicting passage reveals whether the model can distinguish sources, as the legal-research article treats citations as the control.
Three conclusions follow. The retrieved pipeline may win on both reliability and cost, which the reranking article supports, as the evaluation article requires for a decision. A hybrid is possible — retrieve for most questions, use long context for cross-document ones — which the routing article describes. And the result should be recorded, as the decision-memo article requires.
What the test cannot do
It cannot generalise beyond the tested model and question set, which the benchmarks article warns about, as the embedding-selection article advises for own-data evaluation. It does not remove the need for good retrieval, which the RAG article treats as the upstream stage. And it cannot guarantee a future result after a model update, as the monitoring article advises for change triggers.
The counterargument that larger windows make testing unnecessary is wrong in direction, which the lost-middle finding demonstrates: capacity is not use, as Long Context Windows: What a 1M-Token Window Actually Buys You explains.
A decision test
- Fix a question set with known answers, as AAPOR's wording standards advise.
- Move the relevant passage across positions, as Long Context Windows: What a 1M-Token Window Actually Buys You recommends.
- Add distractors with conflicting details, as the hallucination article advises.
- Check citation correctness, as the legal-research article requires.
- Compare with a retrieved baseline, as the embedding-selection article recommends.
- Compute cost per correct answer, as the cost estimator article requires.
- Record the decision, as the decision-memo article requires.
For the mechanism, see Long Context Windows; for the alternative, see Reranking.
References & Further Reading
- Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2023. Lost in the Middle: How Language Models Use Long Contexts — The finding that model performance degrades when relevant information changes position in a long context, and that models do not robustly use long inputs.
- Nils Reimers and Iryna Gurevych, 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — The bi-encoder recipe behind modern semantic search and RAG retrieval.