Retrieved Correctly, Still Wrong? Experimental Design for RAG's "Lost in the Middle"

A RAG system successfully retrieved a passage containing the answer, and that text did enter the model's context window. Yet the final generated answer is still wrong. This failure mode deserves scrutiny—retrieval quality alone cannot explain it; the problem lies in how the model uses the evidence it receives.

This article centers on a testable research question: When the evidence content remains completely unchanged, does merely changing its position in the context alter the answer? Behind this question is the classic study by Liu et al. on long-context usage, Lost in the Middle.

What the Paper Says: Position Affects Utilization Efficiency

Lost in the Middle: How Language Models Use Long Contexts systematically examines how language models handle information at different positions in the input. In two types of experiments—multi-document question answering and key-value retrieval—researchers observed a recurring pattern: when relevant information appears near the beginning or end of the context, model performance is typically best; when the same information is placed in the middle region of a long context, performance degrades.

This finding has direct implications for RAG engineering: stuffing evidence into the context window does not mean the model will reliably use it. However, the original authors explicitly treat this as a hypothesis to be verified, not a universal conclusion that holds for all current models. That is precisely why reproducible experiments are needed to measure one's own system.

Long context window refers to the upper limit of input tokens a language model can receive in a single inference. In recent years, mainstream models' context windows have expanded from thousands of tokens to hundreds of thousands or even millions, making it possible to include multiple documents, entire codebases, or long conversation records in a prompt. Yet window capacity and effective utilization are two different things: the model's attention allocation across different regions of the input is uneven, and the computational and memory overhead of attention mechanisms grows quadratically with sequence length—often mitigated in engineering via positional encoding extrapolation, sliding window attention, or retrieval-based memory. Therefore, "fitting it in" is only the first step; the actual position of evidence within the window directly affects whether the model can reliably recall and reason from it.

Experimental Design: Change Only Position, Not Content

To isolate the variable of "position," the starting point should be question-answer pairs where each question has only one clearly supporting piece of evidence. This way, before introducing multi-evidence questions, the position effect can be measured cleanly on its own.

For each question, construct three versions: the supporting passage placed at the beginning, middle, and end. The question text, original evidence, distractor passages, and instructions all remain unchanged—only the passage order varies. A concrete example:

• Question: "How long does a customer have to request a refund?"
• Supporting evidence: "Refund requests must be submitted within 30 days of purchase."

What needs to be observed: can the model consistently answer "30 days" across all three positions and correctly cite the corresponding passage?

Making Comparisons Statistically Meaningful

A single example proves nothing. The experiment should use an annotated question set, repeat comparisons with multiple different distractor orderings, and cover different context lengths. Within each comparison, the model version, generation parameters, and output budget must be fixed; when passages are reordered, passage IDs must always follow their corresponding text.

There is also an easily overlooked pitfall: you must confirm the input was not truncated. Otherwise, what you measure may be the effect of "evidence loss" rather than "evidence position." Additionally, including a baseline version containing only the supporting passage can identify questions the model fails even without distractors, avoiding misattributing model capability issues to position effects.

Context truncation is the most common confound in long-input experiments. When the concatenated prompt exceeds the model's context window or the server's max_tokens limit, the system typically silently drops content from the tail or head, causing the supporting evidence to fall entirely outside the model's visible range. In this case, the observed accuracy drop reflects "evidence absence," not "evidence in the middle." Before the experiment, explicitly count the tokens of each prompt to ensure it stays below the window limit with room for the generation budget; also record whether server-side truncation occurred, and if necessary, filter or bucket long samples to avoid misreading capacity issues as position effects.

Three Core Metrics

Answer accuracy: whether the answer matches the annotated answer.

Citation correctness: whether the cited passage actually supports the answer. Merely returning an existing passage ID is not enough—the citation must semantically correspond to the answer.

Position sensitivity: how much accuracy varies across the three placements (beginning, middle, end).

Results should be reported separately by model and context length, with uncertainty intervals for estimates. Comparisons without error ranges can hardly support meaningful conclusions.

Uncertainty intervals quantify how much the observed accuracy differences might be due to randomness. For binary correct/incorrect outcomes, Wilson intervals or bootstrap resampling can estimate confidence intervals for accuracy; when comparing beginning versus middle placements, compute difference intervals for paired samples rather than comparing two point estimates in isolation. With small sample sizes, intervals will be wide, and the conclusion that "the middle is worse" may not be robust. Reporting intervals allows readers to judge whether effects are practically meaningful and facilitates comparable contrasts across models and context lengths.

Both Outcomes Are Valuable

If the middle position performs worse, the next experiment can compare different context ordering strategies under the same token budget to see whether this degradation can be mitigated.

If no significant difference appears, that is equally useful information—it means the pattern described in the paper did not clearly replicate under these test conditions, which is itself a finding that needs explanation.

Regardless of the outcome, conclusions must be bounded: limited to the tested models, question sets, and context lengths. Generalization inferences across models require additional evidence.

Final Thoughts

Successful retrieval is only half of RAG answer success. The next question is: can the model reliably use the evidence we provide, and is our context construction helping or hindering it?

It should be noted that this article describes an experimental plan based on research literature; results have not yet been empirically tested. If you have tested the impact of passage order on answers in your own RAG application, you are welcome to share your observations.

Reference: Liu et al., Lost in the Middle: How Language Models Use Long Contexts, https://arxiv.org/abs/2307.03172