LEAPS Benchmark Released: Long-Context Models Ace Recall but Falter Sharply on State Tracking and Pattern Learning

Long-context models (LCLMs) are becoming the core engine of agents—they need to handle the long-term evolution of codebases, run research experiments, and even make predictions based on lengthy historical data. These scenarios demand far more than "find a fact in the context." A new paper points out a critical issue overlooked by existing evaluation systems: models perform nearly perfectly on recall tasks but can fail miserably at state tracking and pattern learning.

Three Long-Context Capabilities, but Existing Benchmarks Only Test the First

The paper decomposes long-context capability into three categories. Recall: the answer exists directly in the context, and the model only needs to locate and extract it. State tracking: the model must apply a series of transformations to a given state and compute the result. Pattern learning: the model must infer a distribution from a large number of observations and then make predictions based on it.

Existing benchmarks almost exclusively focus on recall. This creates a dangerous illusion: if a model scores close to 100% on "needle-in-a-haystack" tasks, its long-context capability is considered adequate. But in real agent workflows, state tracking and pattern learning are the more common sources of cognitive load.

"Needle in a Haystack" is a classic test for evaluating long-context recall: a specific fact (the "needle") is randomly inserted into a long text, and the model is asked to retrieve it when answering a question. Its popularity stems from being easy to construct and score, but its limitations are also obvious—the task is essentially pattern-matching-style information localization, involving no dynamic manipulation of information or multi-step reasoning. Therefore, a near-perfect score on this test only shows that the model's attention mechanism can cover long-distance positions; it does not prove that the model can structurally understand and operate on long contexts. The recall tasks in LEAPS also fall into this category, but its anti-contamination design avoids memorization effects, making the measurement purer.

How LEAPS Is Built: Anti-Contamination, Million-Token Scale, Reliable Metrics

LEAPS includes 12 entirely new datasets covering agent workflows, behavior modeling, market and world-event prediction, and other scenarios. Its design has three core considerations.

Anti-data-contamination is the primary concern. Because the datasets are newly constructed, models cannot have seen this content during pretraining, avoiding the risk of benchmark scores being inflated by memorization effects. Second, input length can scale to millions of tokens, enough to test the limits of current models with the longest context windows. Third, it uses reliable evaluation metrics to ensure that comparisons across models are meaningful.

The paper evaluates 30 API and open-source models, with input token ranges from 8K to 512K. This scale is considerable for同类 studies, covering the full spectrum from lightweight open-source models to frontier closed-source APIs.

Core Finding: Recall Ability Does Not Predict State Tracking or Pattern Learning

The experimental results reveal an unsettling pattern. Models that perform nearly perfectly on recall tasks may show significant divergence in state tracking and pattern learning, and as input length increases, the degradation is far more severe than on recall tasks.

Even more counterintuitive is the performance on pattern learning. From a theoretical perspective, more observations should make inferring the distribution easier—more data should reduce learning difficulty. But the empirical results are exactly the opposite: the longer the input, the worse the model performs on pattern learning. This indicates that current LCLMs have some systematic capability bottleneck when processing statistical regularities in long sequences, rather than a simple "can't find the information" problem.

State tracking tasks require the model, given an initial state, to sequentially apply a series of transformations (such as updates, deletions, and merges) and output the final state. Such tasks simulate the need for agents to maintain an internal world model during long-term operation, such as the version evolution of a codebase or the continuous updating of experimental state. Unlike recall, failures in state tracking are often not "can't find the information" but "found the right information but computed the transformation wrong" or "lost intermediate state." As sequences grow and the number of transformation steps increases, error accumulation becomes significant, causing performance to plummet. This reveals a fundamental shortcoming in current LCLMs' ability to maintain and operate on structured state over long sequences.

The Unexpected Disadvantage of Thinking Models

The paper also reports a noteworthy finding: models with a thinking chain (thinking) often perform worse on LEAPS than their corresponding base models. There are two reasons—thinking models may exhaust the generation length budget, or they may oversimplify the problem when facing complex tasks.

This finding revises the current narrative that "reasoning models are comprehensively superior to base models." In long-context scenarios, additional thinking steps do not necessarily bring gains and may instead introduce new failure modes.

Chain-of-Thought (CoT) is a prompting technique that requires the model to generate intermediate reasoning steps before giving a final answer, aiming to improve accuracy on complex reasoning tasks. However, in long-context scenarios, CoT may have negative effects: first, the generation length budget is limited, and an overly long thinking process may crowd out space for generating the final answer, causing the answer to be truncated or hastily concluded; second, the model may oversimplify complex patterns in long contexts into local heuristics, thereby losing global information. The findings of LEAPS suggest that the benefits of CoT are highly task-dependent, and on tasks requiring long-range state maintenance or distribution inference, its advantage may reverse.

Implications for Long-Context Model Development

The value of LEAPS lies in decomposing long-context capability from a vague holistic concept into independently measurable dimensions and proving that there are significant capability gaps between these dimensions. For teams building long-context agents, this means that when selecting models, they cannot look only at scores on recall-style benchmarks.

The paper's authors hope that LEAPS will guide the development direction of next-generation LCLMs—from "can remember more" to "can correctly reason and generalize over long sequences." This requires architectural innovation, not merely expanding the context window.

One-line Summary

New benchmark LEAPS reveals that long-context models are strong at recall but weak at state tracking and pattern learning, and thinking models perform worse than base models.

Paragraph Summary

The LEAPS benchmark decomposes long-context capability into three categories: recall, state tracking, and pattern learning. It finds that existing evaluations almost exclusively test recall, leading models to score near-perfectly on "needle-in-a-haystack" tasks and be mistakenly considered adequate at long-context capability. LEAPS includes 12 anti-contamination datasets, supports inputs scaling to millions of tokens, and evaluates 30 models (8K–512K tokens). The core finding is that recall performance is decoupled from state tracking and pattern learning, with the latter two degrading more sharply as input grows; in pattern learning, more observations paradoxically lead to worse performance, exposing a systematic bottleneck; models with chain-of-thought often perform worse than base models because they exhaust the generation budget or oversimplify the task. The paper calls for a shift from "can remember more" to "can correctly reason and generalize over long sequences."