Needle in a Haystack
A classic benchmark for evaluating the information retrieval capability of long-context models. It inserts a key piece of information into a long text and asks a question to test whether the model can accurately locate and use that information; accuracy can be plotted as a curve against the position of the information.
Timeline (last 90 days)
"Context rot" refers to the phenomenon where many models nominally offer a 1M window but show a marked drop in recall in the middle of the sequence or under extremely long inputs.
Nominal support for a 1M context does not equal maintaining the same reasoning quality at a 1M length, and the industry commonly uses the "needle in a haystack" test to examine a model's information retrieval and reasoning capabilities in ultra-long contexts.
In a needle-in-a-haystack experiment with a 128K Token context and about 16,000 key-value pairs, changing the position of just one key-value pair produced a maximum accuracy gap of 40.2% for DeepSeek-V4-Flash-Base, 34.8% for DeepSeek-V4-Pro-Base, and still 19.1% and 14.8% respectively after post-training optimization.
Some models show a noticeable drop in needle-in-a-haystack accuracy beyond 32K tokens, while others remain stable over longer ranges.
The Needle in a Haystack test method was first proposed by Greg Kamradt in 2023.
All Facts (5)
"Context rot" refers to the phenomenon where many models nominally offer a 1M window but show a marked drop in recall in the middle of the sequence or under extremely long inputs.
50%UnverifiedNominal support for a 1M context does not equal maintaining the same reasoning quality at a 1M length, and the industry commonly uses the "needle in a haystack" test to examine a model's information retrieval and reasoning capabilities in ultra-long contexts.
50%UnverifiedIn a needle-in-a-haystack experiment with a 128K Token context and about 16,000 key-value pairs, changing the position of just one key-value pair produced a maximum accuracy gap of 40.2% for DeepSeek-V4-Flash-Base, 34.8% for DeepSeek-V4-Pro-Base, and still 19.1% and 14.8% respectively after post-training optimization.
50%UnverifiedSome models show a noticeable drop in needle-in-a-haystack accuracy beyond 32K tokens, while others remain stable over longer ranges.
50%UnverifiedThe Needle in a Haystack test method was first proposed by Greg Kamradt in 2023.
50%