LEAPS
A benchmark for evaluating long-context models. It decomposes long-context capability into three dimensions: recall, state tracking, and pattern learning. It contains 12 contamination-resistant datasets and supports inputs of up to one million tokens.
Timeline (last 90 days)
LEAPS is a long-context evaluation benchmark comprising 12 entirely new datasets, covering scenarios such as agentic workflows, behavior modeling, and market and world event forecasting.
30 API and open-source models were evaluated on LEAPS, with input token ranges from 8K to 512K.
On LEAPS, models with near-perfect performance on recall tasks may diverge significantly on state tracking and pattern learning, with degradation becoming more severe as input length increases.
On LEAPS's pattern learning tasks, the longer the input, the worse the model performs.
Reasons why thinking models perform worse than base models on LEAPS include exhausting the generation length budget or over-simplifying problems on complex tasks.
LEAPS adopts a data-contamination-resistant design, with datasets built entirely anew to avoid inflated scores due to models' pretraining memory.
The paper's authors hope that LEAPS can guide the development direction of next-generation LCLMs, shifting from "being able to remember more" to "being able to reason and generalize correctly over long sequences".
All Facts (7)
LEAPS is a long-context evaluation benchmark comprising 12 entirely new datasets, covering scenarios such as agentic workflows, behavior modeling, and market and world event forecasting.
50%Unverified30 API and open-source models were evaluated on LEAPS, with input token ranges from 8K to 512K.
50%UnverifiedOn LEAPS, models with near-perfect performance on recall tasks may diverge significantly on state tracking and pattern learning, with degradation becoming more severe as input length increases.
50%UnverifiedOn LEAPS's pattern learning tasks, the longer the input, the worse the model performs.
50%UnverifiedReasons why thinking models perform worse than base models on LEAPS include exhausting the generation length budget or over-simplifying problems on complex tasks.
50%UnverifiedLEAPS adopts a data-contamination-resistant design, with datasets built entirely anew to avoid inflated scores due to models' pretraining memory.
50%UnverifiedThe paper's authors hope that LEAPS can guide the development direction of next-generation LCLMs, shifting from "being able to remember more" to "being able to reason and generalize correctly over long sequences".
50%