Needle in a Haystack
A long-context evaluation method for large language models proposed by Greg Kamradt in 2023. It inserts a target sentence at various positions within a long document and tests the model's retrieval ability, producing a two-dimensional heatmap to assess retrieval success rates across different context positions.
Core Facts
Timeline (last 90 days)
The Needle-in-a-Haystack test randomly inserts an irrelevant target sentence at different positions within a long text, and by querying the model draws a two-dimensional heatmap of 'position × context length' to present retrieval success rates.
Single-hop needle-in-a-haystack retrieval, contaminated benchmarks from years ago, and few-shot in-context learning are the three evaluation scenarios most favorable to compression and sparsity methods, capable of making almost any method look good.
The Needle in a Haystack test was proposed by Greg Kamradt in late 2023, originally using Paul Graham's essays as filler text, with the 'needle' being a statement about the best pizza place in San Francisco.
In single-hop needle-in-a-haystack tasks, critical KV pairs are naturally prominent in attention scores, making it nearly impossible for compression methods to drop them, making these tasks too 'friendly' to compression methods.
The Needle in a Haystack test was proposed by Greg Kamradt in late 2023, originally using Paul Graham's essays as filler text.
Single-hop needle-in-a-haystack tests, contaminated benchmarks, and few-shot in-context learning tasks are the three categories most favorable to compression and sparse methods, with most being passable under sliding window attention.
The Needle in a Haystack test was originally proposed by Greg Kamradt in late 2023 to evaluate a model's ability to precisely retrieve specific information in long contexts.
All Facts (7)
The Needle in a Haystack test was originally proposed by Greg Kamradt in late 2023 to evaluate a model's ability to precisely retrieve specific information in long contexts.
80%UnverifiedThe Needle-in-a-Haystack test randomly inserts an irrelevant target sentence at different positions within a long text, and by querying the model draws a two-dimensional heatmap of 'position × context length' to present retrieval success rates.
50%UnverifiedSingle-hop needle-in-a-haystack retrieval, contaminated benchmarks from years ago, and few-shot in-context learning are the three evaluation scenarios most favorable to compression and sparsity methods, capable of making almost any method look good.
50%UnverifiedThe Needle in a Haystack test was proposed by Greg Kamradt in late 2023, originally using Paul Graham's essays as filler text, with the 'needle' being a statement about the best pizza place in San Francisco.
50%UnverifiedIn single-hop needle-in-a-haystack tasks, critical KV pairs are naturally prominent in attention scores, making it nearly impossible for compression methods to drop them, making these tasks too 'friendly' to compression methods.
50%UnverifiedThe Needle in a Haystack test was proposed by Greg Kamradt in late 2023, originally using Paul Graham's essays as filler text.
50%UnverifiedSingle-hop needle-in-a-haystack tests, contaminated benchmarks, and few-shot in-context learning tasks are the three categories most favorable to compression and sparse methods, with most being passable under sliding window attention.
50%