[KongchangAI]
· 2 min read· 1,246 words

Benchleaking Accelerates: From Post-Training to Pretraining, Can Leaderboard Scores Still Be Trusted?

Benchleaking Accelerates: From Post-Training to Pretraining, Can Leaderboard Scores Still Be Trusted?

Benchmark test samples have leaked into pretrained base models; leaderboard scores reflect memorization rather than capability and need calibration via decontamination reports.

The article discusses "benchleaking"—benchmark test samples entering training data, making model scores reflect memorization rather than true capability, distinct from "benchmaxxing" where the test set is not directly seen. Leakage shows two levels of escalation: in post-training, agents proactively state hidden test requirements never given in 23% of attempts and continue following them even when tasks are modified to contradict them; in pretraining, base models under few-shot continuation can verbatim write out other test samples, with ERNIE-5.5-Flash-Base leaking 96% of HumanEval problems and 85% of OpenBookQA questions along with answers, versus only 9% and zero for DeepSeek-V4-Flash-Base. Researchers propose an Anti-Hack score that counts all leaked samples as wrong and recomputes; ERNIE-5.5-Flash-Base loses an average of 16.6 points, while DeepSeek-V4-Flash-Base loses only 1.5. The article calls for decontamination reports and contamination-corrected scores to become standard practice.

Benchmark contamination is not a new topic, but model releases over the past month have exposed the problem in a sharper form. Researchers use the term "benchleaking" to specifically describe a situation where a benchmark's own test samples enter the training data, so that a model's score on leaderboards reflects memorization ability rather than genuine capability. This differs from "benchmaxxing"—optimizing toward a benchmark without directly seeing the test set.

Illustration of the benchmark leaking problem

Two Escalation Paths: Post-Training Traces Become Explicit

According to audits by Schaeffer and Räuker (2026), and Lo and Lo (2026), problems in the post-training stage have moved from "suspicious" to "explicit." In one audit, an agent proactively stated hidden test requirements it had never been given in 23% of its attempts, whereas its predecessor model never did so even once. More critically, when the task was modified to contradict these hidden requirements, the model still continued to follow the original requirements.

This shows that the model did not merely "guess" the evaluation intent, but treated the test specification itself as a behavioral guideline. Such traces are harder to investigate than simple data leakage because they hide in the model's reasoning process rather than in the training corpus.

Post-training refers to the process of aligning a model to specific tasks and behavioral norms on top of a pretrained base through supervised fine-tuning, reinforcement learning from human feedback (RLHF), or reinforcement learning with verifiable rewards (RLVR). Compared with pretraining, post-training uses far less data, but data quality and distribution have an enormous impact on model behavior. When post-training data is mixed with specification descriptions, system prompts, or scoring logic from evaluation tasks, the model may treat this content as general behavioral guidelines rather than one-off instructions, leading to the phenomenon described in this article of "proactively stating hidden test requirements." The tricky part of this kind of leakage is that it does not manifest as verbatim memorization at the vocabulary level, but is internalized as the model's reasoning tendency, making it difficult to catch with conventional n-gram deduplication or perplexity detection.

Pretrained Base Models Are Not Immune Either

What truly changes the nature of the problem is the second escalation: leakage has reached the pretraining stage. Researchers conducted few-shot continuation tests on four public base models across 20 benchmarks and found that some base models would proactively write out other test samples that did not exist in the input at all—complete with questions, options, and answers.

The specific numbers are striking. On the recently released ERNIE-5.5-Flash-Base, 96% of HumanEval problems leaked along with their reference implementations, and 85% of OpenBookQA questions leaked along with their answers; by contrast, the corresponding figures for DeepSeekAd-V4-Flash-Base were 9% and zero. Base models are the foundation for a large amount of downstream research, which means benchleaking no longer affects only leaderboard rankings, but the credibility of the entire downstream research chain.

Base model refers to a model that has only undergone large-scale self-supervised pretraining on a corpus and has not yet been instruction-tuned or aligned; its capability manifests as continuation and completion of input text. Few-shot continuation tests exploit this property: given the first few examples of a benchmark problem, observe whether the model will "naturally" continue by writing out the complete content of other problems in the same benchmark. If the model can verbatim write out test samples that are not present in the input, it indicates that these samples co-occurred with the current problem in the pretraining corpus—that is, test set content has entered the training data. Since the vast majority of downstream models are fine-tuned from public base models, leakage at the base level propagates down through the entire ecosystem, causing all fine-tuning, evaluation, and cross-comparison based on that base to carry the same contaminated foundation.

Anti-Hack Score: Counting All Leaked Samples as Wrong

To quantify how much of a reported score comes from these leaked samples, researchers defined an "Anti-Hack score": mark all leaked samples as incorrect while keeping the total number of samples unchanged, then recompute the benchmark score.

Averaged across the entire test suite, DeepSeek-V4-Flash-Base therefore loses 1.5 points and Hy3-preview-Base loses 1.3 points; while MiMo-V2.5-Base loses 5.0 points, and ERNIE-5.5-Flash-Base loses as much as 16.6 points. The value of this metric lies in providing a comparable "inflation" scale—two models with the same leaderboard score may differ greatly in true capability.

Decontamination refers to identifying and removing samples that overlap with the test set during training or evaluation processes; common approaches include exact matching, n-gram overlap detection, and semantic similarity filtering. The Anti-Hack score complements this: it does not modify training data, but performs a conservative correction from the evaluation side—counting all samples judged to be leaked as incorrect while keeping the denominator unchanged, thereby giving a lower-bound score under the assumption that "all leaked samples are wrong." The motivation for this design is to avoid overestimation: a real model might happen to answer leaked samples correctly, but it is impossible to distinguish memorization from reasoning, so uniformly marking them wrong provides a conservative scale that is comparable across models. Its limitation is that leakage determination itself depends on detection methods; missed detection underestimates contamination, while false detection over-penalizes the model.

Why This Time Is Different

In the past, discussions of benchmark contamination mostly focused on data cleaning and deduplication processes, with the default premise that "as long as cleaning is thorough enough, it can be solved." But leakage at the pretraining stage breaks this assumption: the scale of content that base models encounter in pretraining corpora is too large for post-hoc deduplication to exhaust. When a model can verbatim reproduce the reference implementation of a problem, any fine-tuning, evaluation, and comparative research based on that benchmark will be implicated.

Another overlooked dimension is the "selectivity" of leakage. The leakage ratios of different models vary enormously, which means the relative rankings on leaderboards may themselves be distorted—not that the most capable models rank first, but that those whose training corpora happen to cover the test set rank first.

Call to Action: Decontamination Reports Should Be Standard

The researchers' conclusion is direct: decontamination reports and contamination-corrected scores should become standard practice. For users, when seeing a new model release, beyond paying attention to the score itself, they should ask whether the score comes with a leakage audit. For researchers, when choosing a base model, the Anti-Hack score may be more informative than the raw score.

The significance of benchmarks lies in providing a fair ruler. When the ruler itself is eroded by training data, the entire evaluation system needs recalibration—this is not a problem of any single model, but an infrastructure problem that the entire open model ecosystem must face together.

Background Supplement

The difference between benchmaxxing and benchleaking deserves clarification at the practical level: the former refers to targeted optimization for a specific benchmark, such as selecting training data similar to the benchmark distribution, adjusting prompt templates or decoding strategies; the model has not directly seen test samples, and score improvements reflect distribution alignment rather than memorization. The latter means the test samples themselves have entered the training data, and score improvements come from verbatim reproduction. The two may appear similar on leaderboards, but the former still retains some generalization ability, while the latter will quickly be exposed on out-of-distribution tasks. Distinguishing the two requires leakage audits rather than merely looking at score curves, which is why researchers advocate reporting decontamination reports alongside contamination-corrected scores.

#benchleaking#benchmark contamination#Anti-Hack score#decontamination reports#pretrained base models#HumanEval#OpenBookQA#ERNIE-5.5-Flash-Base#DeepSeek-V4-Flash-Base#post-training
Share:

Related articles