Lost in the Middle: Why LLMs Miss Critical Information in Long Prompts — and What to Do About It

Paste a 40-page policy document into a prompt, ask a question, and the model confidently answers as if page 19 simply doesn't exist. The fact you need is right there — it just happens to land in the middle.

This isn't bad luck. It's a repeatedly measured and validated phenomenon called "Lost in the Middle." The single most important takeaway is simple: put your strongest signal at the beginning and end of a long prompt — never only in the middle.

What the Research Actually Found

In the paper Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2024), researchers from Stanford, UC Berkeley, and Samaya AI ran a clean experiment. They gave a model a question along with 10, 20, or 30 retrieved Wikipedia passages, exactly one of which contained the answer. Then they moved that answer passage around — to the beginning, somewhere in the middle, or the end — while keeping everything else constant.

If models used context uniformly, accuracy would be flat regardless of position. Instead, results showed a clear U-shaped curve:

• Beginning of context: highest accuracy — what the paper calls primacy bias.
• End of context: second highest — recency bias.
• Middle: a notable dip.

The specific numbers are striking. In the 20-document setting, GPT-3.5-Turbo answered correctly about 75.8% of the time when the relevant passage was first, about 63.2% when it was last, and only 53.8% when it fell in the middle. Its "closed-book" score — with no documents at all — was 56.1%. In other words, at the worst position, giving the model the answer actually performed slightly worse than giving it nothing.

Two additional findings are equally important for developers:

• Larger context windows don't fix the problem. Extended-context versions (16K, 100K) performed almost identically to shorter-window versions when both could fit the same input. A larger window means you can send more — not that the model uses it well.
• Retrieving more documents quickly hits a ceiling. In open-domain QA, increasing retrieved documents from 20 to 50 improved GPT-3.5-Turbo by only ~1.5% (Claude-1.3 by ~1%), at the cost of significantly more tokens and latency.

Why It Happens

The follow-up paper Found in the Middle (Hsieh et al., ACL Findings 2024) attributes this effect directly to the model's attention mechanism: tokens at the beginning and end of input receive more attention regardless of their actual relevance. Their calibration approach reduced this bias and improved RAG results by up to 10 percentage points over prior methods.

Think of it like a long meeting. People remember how it opened and how it closed. The important point raised at minute 34? That's the one nobody wrote down.

Newer models perform much better than the 2023-era models in the paper on simple "needle in a haystack" tests. But in real multi-document tasks, the position effect hasn't disappeared. Treat placement as an actively designed variable — not something to leave to chance.

A note on the attention mechanism: The Attention Mechanism is the core component of the Transformer architecture. It allows a model to assign different weights to different positions in the input sequence when generating each output token. In theory, attention heads can freely "attend" to any position — token 1 or token 10,000. In practice, however, the distribution of pretraining data shapes the attention patterns the model learns: most training text (books, web pages, code) concentrates key information at the beginnings and ends of paragraphs, biasing the model at inference time to "attend" more heavily to those positions. Positional encoding design can also reinforce this bias — relative positional encodings often have weaker discriminability for middle positions in very long sequences, making the identity signal for mid-sequence tokens fuzzier. This structural bias isn't a bug in any single model; it's a systemic characteristic shared across the Transformer family under current training paradigms.

Before and After

Here's a typical long prompt, stacked in the order the data arrived:

You are a support assistant.

<documents>
  [doc 1: general company FAQ]
  [doc 2: shipping policy]
  [doc 3: refund policy  <- answer is here]
  [doc 4: warranty terms]
  [doc 5: holiday hours]
</documents>

Answer the customer's question. Only use the documents.
Keep it under 80 words. Cite the doc number.

Question: Can I get a refund after 45 days?

The critical rules ("only use the documents," word limit) and the one document that actually matters are both buried in the middle of the document stack.

Here's the same content, deliberately repositioned:

You are a support assistant. Answer only from the documents below.
If the answer isn't in them, say so. Keep it under 80 words. Cite the doc number.

Task: answer a customer question about refunds.

<documents>
  [doc 3: refund policy]      <- most relevant first
  [doc 4: warranty terms]
  [doc 1: general company FAQ]
  [doc 5: holiday hours]
  [doc 2: shipping policy]    <- second most relevant last
</documents>

Question: Can I get a refund after 45 days?
Reminder: answer only from the documents, under 80 words, cite the doc number.

Rules go at the top. The question and a one-line restatement of the rules go at the bottom. The strongest content blocks sit at the two ends where model attention is most concentrated.

This aligns closely with Anthropic's long-context guidance — they recommend placing long documents above the query and note that putting the query at the end improved response quality by up to 30% in their tests, particularly on complex multi-document inputs.

The RAG Pitfall: Ranked ≠ Placed Correctly

Most RAG pipelines retrieve top-k chunks, sort them by score, and stuff them in directly. This puts chunk #1 at the top — fine — but when you have 10+ chunks, the equally important chunks #2 through #4 end up in the middle.

Better approaches:

• Rerank first, then trim. Use a reranker and keep fewer chunks. Five good chunks beat twenty mediocre ones.
• Place your best chunks at the edges. Put rank 1 first, rank 2 last, rank 3 second, rank 4 second-to-last — let the weakest ones fall into the middle. LangChain implements this as LongContextReorder; rolling your own takes about ten lines of code.
• Don't pad the context. Every extra chunk you add pushes other content toward the middle.

def edge_order(chunks_by_score):
    """Best chunks at the edges, weakest in the middle."""
    front, back = [], []
    for i, chunk in enumerate(chunks_by_score):
        (front if i % 2 == 0 else back).append(chunk)
    return front + back[::-1]


# [r1, r2, r3, r4, r5] -> [r1, r3, r5, r4, r2]

A note on RAG terminology: RAG (Retrieval-Augmented Generation) is an architectural paradigm that combines an external knowledge base with a language model: the system first retrieves the most relevant text chunks from a vector database or search engine, then inserts those chunks into the prompt so the model can generate answers grounded in the retrieved content. A Reranker is an optional precision layer in the RAG pipeline — typically a fine-tuned cross-encoder that scores query–document pairs with finer-grained relevance than approximate nearest-neighbor vector retrieval, at higher computational cost. A common two-stage engineering pattern is to use vector retrieval to quickly recall top-50 candidates, then rerank to top-5. LongContextReorder is a built-in LangChain utility implementing exactly the findings of the "Lost in the Middle" paper: it takes a reranker's ordered list and interleaves it to the head and tail of the prompt, ensuring the highest-relevance chunks naturally land where model attention is strongest.

Quick Checklist

• Instructions and hard constraints go at the top
• Restate the question or task at the end
• For RAG, place the most relevant chunks near the beginning or end — don't just pile them in order by score
• Rerank and keep fewer chunks
• Keep context as short as the task allows
• Test it empirically: move the key fact to the beginning, middle, and end of your real prompt and observe whether the model's answer changes

That last one takes ten minutes and tells you more about your specific model than any benchmark.

Closing Thoughts

"Lost in the Middle" is a reminder that no matter how large a model's context window is, it doesn't mean every token inside that window gets equal treatment. Position is an engineering variable you can actively design for: put high-signal content at the edges, rerank and trim your retrieval results, and validate with a simple A/B test. These low-cost adjustments consistently deliver more tangible gains than blindly expanding context length or piling in more documents.