Benchmark Contamination Inflates LLM Scores but Rarely Disrupts Leaderboard Rankings

Benchmark Contamination: A Misunderstood Threat

Evaluating large language model (LLM) capabilities relies heavily on benchmark leaderboards — scores on ARC, GSM8K, HellaSwag, MMLU, and similar benchmarks have become the industry's de facto yardstick for comparing models. Yet a problem has long troubled researchers: benchmark contamination.

Benchmark contamination occurs when test-set questions leak into training data. If a model has already "seen" the exam questions during training, it will naturally score higher. This has been widely viewed as a serious threat to leaderboard credibility. But a new study published on arXiv (arXiv:2609.02899) offers a more nuanced perspective: we've been conflating two fundamentally distinct questions all along.

The authors argue that concerns about contamination actually carry two separate meanings: first, whether contamination inflates absolute scores; and second, whether contamination changes the relative ranking order among models. The answers to these two questions, it turns out, can be very different.

Measuring "Memorization" vs. "Ability" in LLMs

The most elegant part of this research lies in how it reframes benchmark contamination as a violation of anchor-item invariance.

Original vs. Semantically Equivalent Rewritten Questions

The core method compares model performance on original questions against semantically equivalent rewritten versions. The logic is straightforward: if a question is rewritten without changing its meaning, a model that truly "understands" the underlying knowledge should perform equally well on both versions. A model that has merely "memorized" the original answer, however, will perform unusually well on the original but stumble on the rewrite.

This within-item contrast design is elegant because it holds the measured skill constant, cleanly separating "memorization" from "genuine capability." This is far more rigorous than simply observing absolute scores, since it naturally controls for item difficulty and baseline model ability.

Calibrating the Measurement Tool with Known Contamination Doses

Rather than jumping to conclusions, the research team first rigorously calibrated their measurement approach. They analyzed item-level response data from two sets of models:

• 47 publicly released models
• 74 models fine-tuned with known contamination doses

This covered four major benchmarks: ARC, GSM8K, HellaSwag, and MMLU. The results showed that the measurement method recovers injected contamination in a dose-responsive manner — the corrected effect of test-set leakage was +0.187 accuracy percentage points. Crucially, for "negative control" models trained only on legitimate data, the method never falsely flagged contamination (effect size: -0.012), landing almost perfectly at zero. This validates the reliability of the measurement tool itself.

Core Finding: Scores Inflate, but LLM Leaderboard Rankings Hold Steady

With calibration complete, the study moves to its central question — quantifying contamination's actual impact on leaderboards.

Rank Correlation of 0.997

The study compared rank correlations between standard leaderboards and "rewrite-controlled leaderboards," finding a correlation of 0.997. This figure is nearly 1, meaning that after removing contamination effects, model rankings remain almost entirely unchanged.

Sensitivity analyses further corroborate this: the degree of "differential contamination" actually observed falls far below the level needed to meaningfully shift rankings. Across 188 model × benchmark combinations, only 3 cases showed differential contamination verified by two reference points.

The Critical Distinction: Uniform vs. Differential Contamination

This finding reveals a mechanism that has been largely overlooked. The contamination affecting publicly available models is largely uniform — meaning nearly all models are "tainted" by test data to a similar degree. When contamination is uniform, it raises the absolute scores of all models simultaneously, like across-the-board inflation: the numbers go up, but the relative standings don't change.

What actually distorts rankings is differential contamination: when some models are far more contaminated than others. And this, in practice, is quite rare — appearing in only 3 out of 188 cases. In other words, leaderboard distortion only occurs in these low-probability, extreme scenarios.

Practical Implications for LLM Evaluation and Model Selection

The value of this research isn't in exonerating benchmark contamination — it's in correctly locating the problem.

First, the study acknowledges that contamination is real and does inflate absolute scores. When we see a model score 90% on MMLU, some of that may be due to memorization, and absolute numbers should be taken with a grain of salt.

Second, and more importantly, it demonstrates that relative comparisons based on leaderboards remain reliable. If you want to know whether model A outperforms model B, existing leaderboard rankings are unlikely to be wrong due to contamination. This is good news for practitioners who rely on leaderboards for technical decision-making.

Based on these findings, the authors offer concrete, actionable recommendations:

1. Use calibrated invariance auditing tools — the research has open-sourced a reference implementation for the community to examine model contamination;
2. Leaderboards should report rewrite-controlled rankings alongside confidence intervals, rather than only presenting single absolute scores;
3. Focus monitoring on differential contamination — concentrate attention on identifying individually anomalous models that are disproportionately contaminated, as these represent the true risk.

Conclusion: Toward a More Mature View of AI Model Evaluation

In today's white-hot LLM race, leaderboard credibility underpins the entire industry's judgment. This study, through rigorous experimental design, provides a quantitative answer to a long-inflated concern: contamination makes scores look bigger, but rarely reshuffles the rankings. It reminds us that when evaluating AI models, rather than fixating on the absolute accuracy of absolute scores, we should focus on the robustness of relative rankings — and on the rare cases of differential contamination that can genuinely undermine fairness.

This may also represent a more mature philosophy of AI evaluation: acknowledge that measurement is imperfect, but use scientific methods to define the boundaries of that imperfection.