Decontaminated Dataset Still Contains 52 HumanEval Problems: Benchmark Contamination Is Underestimated

Training a 360M-parameter code model, we reported 26.8% pass@1 on HumanEval — two days later, a reader asked whether we had checked for overlap between the instruction data and the benchmark. We hadn't. After checking, the honest number was 22.6%.

This is a public self-correction from the trifero.ai lab at Strife Technologies. They found that even when using a dataset labeled as "decontaminated," 52 of HumanEval's 164 problems were still lurking inside.

Scanning Method: Two Gates

The research team ran two checks on each instruction set:

• Problem text: Take the first 80 characters of each HumanEval docstring, normalize whitespace, and compare.
• Reference solution: Strip comments and docstrings, then check whether more than half of the 13-token n-grams in the solution match.

Dataset	Samples	Docstrings Found	Solutions Found
evol-codealpaca-v1 (actually used)	111,272	60 problems, 156 instances	8 problems, 67 instances
Magicoder-Evol-Instruct-110K ("decontaminated" version)	111,183	52 problems, 116 instances	8 problems, 66 instances
Magicoder-OSS-Instruct-75K (actually used)	75,197	0	1 problem, 1 instance

The key point: that filtered and redistributed "decontaminated" version differs from the original by only 89 lines, yet still contains 52 of the 60 contaminated problems. In other words, switching to that version wouldn't have saved you.

n-gram matching is one of the most commonly used approximate duplicate detection techniques in data contamination detection. The basic idea is: split text into fragments of n consecutive tokens (e.g., a 13-gram means every 13 consecutive tokens form one fragment), then count the proportion of shared n-grams between candidate samples and reference samples. The reason for using n-grams rather than exact string matching is that it can catch slight paraphrasing, variable name substitution, or whitespace differences — but these changes still preserve the underlying token sequence structure. If more than half of the 13-grams in an answer can be found in the benchmark reference solution, the two are almost certainly derivative. The choice of n is a tradeoff: too small (e.g., 3) produces many false positives from common phrases; too large (e.g., 50) is overly sensitive to legitimate code rewrites and prone to missed detection. 13 is an empirically reasonable middle ground in code scenarios.

Why the Filter Missed Them

These matches aren't copy-paste. They're benchmark problems in the form of practice sheets: reworded, with extra requirements added, or ported to Java, each with a complete solution attached.

For example, HumanEval/43's pairs_sum_to_zero appears with the original docstring, doctests, and a usable solution, just with an added instruction to "make it linear time complexity."

This isn't an accident of the collection process. Evol-Instruct expands datasets by repeatedly rewriting seed problems, and some seeds are benchmark problems. Exact-match filters can't catch rewrites — and rewriting is the entire point of the method.

Evol-Instruct is a method for expanding instruction datasets by having a large language model repeatedly rewrite seed instructions. The typical process is: given a seed instruction (possibly from an open-source dataset or human-written), have the model generate a "deeper," "broader," or "more complex" new instruction along with a corresponding response. This iterative evolution can quickly produce a large number of diverse samples, but there's a structural risk: if the seed pool contains benchmark problems, then all variants evolved from that seed will inherit its core problem structure, just with different wording or added constraints. Exact-match filters can only find text that's identical to benchmark problems, while Evol-Instruct's outputs are already completely different literally — what they share is semantics and solution structure, not strings. Therefore, a filter designed for exact matching produces almost no interception when facing evolved data.

Is the Model Memorizing Answers?

No. This is precisely what surprised the researchers most.

They compared each chat model output against the reference solution using difflib ratio (calculated line by line after whitespace normalization). Memorized answers would score above 0.9. Of the 44 passes, the highest was only 0.73, and that was a problem whose reference solution was only one line. Most were below 0.3.

So they removed contaminated samples and retrained, while also retraining on the original data with just a different random seed, to distinguish data effects from rerun effects.

Checkpoint	HumanEval	HumanEval+	MBPP
Base	13.4%	11.0%	20.2%
Chat, as published	26.8%	25.6%	38.9%
Same data, seed 2	26.2%	25.6%	34.6%
After decontamination	22.6%	21.3%	35.0%

Decontaminated vs. published: HumanEval -4.3 points, losing 8 problems and gaining 1, McNemar test p = 0.039. The two original-data runs differed by no more than 1 problem, so the drop comes from the data itself, not rerun variance.

difflib is a module in the Python standard library for sequence comparison, and its SequenceMatcher can compute a similarity ratio (ranging from 0 to 1) between two sequences. In code scenarios, researchers typically split model outputs and reference solutions line by line before comparing — the higher the ratio, the closer the two are. If the model is verbatim-reciting an answer seen during training, the ratio typically exceeds 0.9; when the ratio is below 0.3, it means there's almost no line-level overlap between the output and the reference solution. Note that difflib ratios are sensitive to whitespace, comments, and variable naming in code, so normalization is usually applied beforehand. The advantage of this metric is that it's simple and intuitive, but it cannot distinguish between "understood the logic and implemented it differently" and "only remembered fragments" — which is why researchers combined subset analysis to further rule out the memorization mechanism.

Where the Points Were Lost Reveals the Truth

If it were memorization, the lost points should be concentrated in the 63 problems matching removed training data. But that's not what happened.

Subset	Problems	Base	Published	Seed 2	After Decontamination
In removed data	63	5	14	17	13
Not in removed data	101	17	30	26	24

Comparing against the average of the two original-data runs, the decontaminated model scored about 4 points lower on both subsets.

This shows: the contamination is real, but the mechanism isn't memorization. Practice on HumanEval-style problems broadly improved the model's performance on HumanEval-style problems — including problems that never appeared in the data. Researchers had previously reported this improvement as "capability."

McNemar's test is a statistical test for paired binary data, particularly suited for comparing performance differences of the same set of samples under two different conditions. In model evaluation, it's often used to determine whether the pass/fail patterns of two models (or two versions of the same model) on the same benchmark have changed in a statistically significant way. Specifically, it constructs a 2×2 contingency table, counting the number of problems where "version A passes and version B fails" and where "version A fails and version B passes," then tests whether these two counts are significantly asymmetric. Compared to a simple pass-rate difference, McNemar's test has the advantage of focusing on the direction of change per problem rather than the overall mean, thus eliminating interference from different problem difficulty distributions. A p-value less than 0.05 is typically considered a significant difference. In this article, p = 0.039 means the 4.3-point drop from decontamination is unlikely to be due to random fluctuation alone.

MBPP Teaches a Different Lesson

Only 10 MBPP problems matched removed data, while the two original-data runs differed by 11 problems. The published 38.9% is best understood as a lucky random seed. Run-to-run variance on MBPP is about 4 points, comparable to the contamination effect size on HumanEval.

A single run reports one number, and that number simultaneously represents the model, the data, the evaluation framework, and the random seed.

Four Recommendations for Evaluators and Vendors

Scan your instruction data. Docstring prefixes plus solution n-grams take just minutes to run, and the code is open-sourced in the repository.

Don't treat "decontaminated" as a trustworthy label. Check what the filter actually matched. Against a dataset built through rewriting, an exact-match filter is almost equivalent to no filter.

Run random seed controls. Without this control, you can't separate a 4-point data effect from a 4-point variance effect — and on MBPP, the variance effect is actually the larger one.

Ask vendors (including us) three questions: What's the decontamination method? What's the evaluation framework? How many runs are behind the number?

The full report (with charts, confidence intervals, and McNemar tests) is at trifero.ai/research/elroy, and the code along with nine sections and all benchmark completion and failure records are open-sourced under Apache-2.0 at github.com/strifero/elroy.

One-line Summary

trifero.ai found that HumanEval data contamination inflated scores by 4.3 points, but the mechanism isn't memorization — it's style generalization.

Paragraph Summary

The trifero.ai lab, after a reader's challenge, re-examined its training data and found that even the "decontaminated" Magicoder-Evol-Instruct-110K still contained 52 HumanEval problems. After removing contaminated samples, HumanEval pass@1 dropped from 26.8% to 22.6%. However, the lost points weren't concentrated in the removed problems but were evenly distributed across all HumanEval-style problems, indicating that the contamination effect comes not from answer memorization but from the model's generalized practice on similar-style problems being misreported as "capability." Additionally, run-to-run variance on MBPP is about 4 points, comparable to the contamination effect, showing that single-run results are unreliable. The researchers recommend scanning instruction data, questioning decontamination labels, running random seed controls, and pressing vendors for evaluation details.