RULER
A long-context evaluation suite introduced by NVIDIA. It contains a variety of synthetic tasks, including retrieval, multi-hop tracing and aggregation, and supports custom context lengths such as 4K to 128K. It is widely used to measure how effectively a model uses information across long texts.
Timeline (last 90 days)
On six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
The RULER benchmark was proposed by NVIDIA and includes various synthetic tasks such as retrieval, multi-hop tracing, and aggregation, supporting context lengths from 4K to 128K.
Latent Compaction demonstrates advantages on benchmarks such as RULER, LongBench, and Phonebook Lookup.
Latent Compaction achieves 96.7% accuracy on the RULER benchmark at a 32x compression rate, while KVZip+ achieves only 40.0%.
All of SubQ Snap's performance data comes from Subquadratic itself and has not been independently tested by an external lab, and its long-context tests are its own modified versions of RULER and LCR.
Benchmark tests commonly used to evaluate long-memory systems include RULER, NIAH (Needle-in-a-Haystack), and LongBench.
RULER is a benchmark designed specifically to evaluate long-context model capabilities, covering tasks such as single-hop and multi-hop retrieval, question answering, summarization, and verbatim copying, typically evaluated at context lengths of 16K to 128K.
Aggregate metrics (such as multi-task average scores) have an inherent ability to dilute failures; when simpler tasks constitute the majority, the average score can mask significant degradation on a small number of high-difficulty tasks.
RULER is a long-context evaluation benchmark proposed by NVIDIA's research team in 2024, comprising 13 synthetic tasks.
RULER is a long-context evaluation benchmark proposed by the NVIDIA research team in 2024, containing 13 synthetic tasks divided into four categories: NIAH, Variable Tracking, Common/Frequent Words Extraction, and question answering based on SQuAD/HotpotQA.
2 more timeline events
All Facts (12)
On six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
50%UnverifiedThe RULER benchmark was proposed by NVIDIA and includes various synthetic tasks such as retrieval, multi-hop tracing, and aggregation, supporting context lengths from 4K to 128K.
50%UnverifiedLatent Compaction demonstrates advantages on benchmarks such as RULER, LongBench, and Phonebook Lookup.
50%UnverifiedLatent Compaction achieves 96.7% accuracy on the RULER benchmark at a 32x compression rate, while KVZip+ achieves only 40.0%.
50%UnverifiedAll of SubQ Snap's performance data comes from Subquadratic itself and has not been independently tested by an external lab, and its long-context tests are its own modified versions of RULER and LCR.
50%UnverifiedBenchmark tests commonly used to evaluate long-memory systems include RULER, NIAH (Needle-in-a-Haystack), and LongBench.
50%UnverifiedRULER is a benchmark designed specifically to evaluate long-context model capabilities, covering tasks such as single-hop and multi-hop retrieval, question answering, summarization, and verbatim copying, typically evaluated at context lengths of 16K to 128K.
50%UnverifiedAggregate metrics (such as multi-task average scores) have an inherent ability to dilute failures; when simpler tasks constitute the majority, the average score can mask significant degradation on a small number of high-difficulty tasks.
50%UnverifiedRULER is a long-context evaluation benchmark proposed by NVIDIA's research team in 2024, comprising 13 synthetic tasks.
50%UnverifiedRULER is a long-context evaluation benchmark proposed by the NVIDIA research team in 2024, containing 13 synthetic tasks divided into four categories: NIAH, Variable Tracking, Common/Frequent Words Extraction, and question answering based on SQuAD/HotpotQA.
50%UnverifiedPapers can conceal performance degradation on difficult subtasks such as NIAH-MK3 by reporting only aggregated average scores.
50%UnverifiedAmong RULER's 13 tasks, 6 NIAH tasks exhibit cooperative characteristics, 2 QA tasks use datasets from years ago, and the VT task contains a large amount of irrelevant context.
50%