Unverified50% confidenceFactExact time
After decontamination, the model achieved 22.6% on HumanEval, 21.3% on HumanEval+, and 35.0% on MBPP; compared with the published version, HumanEval dropped by 4.3 points, McNemar test p = 0.039.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/13/2026
First Seen
Valid until: 1/11/2027
Sources
Related Entities
Related Claims
UnverifiedThe decontaminated model was about 4 points lower than the mean of the two original-data runs on both the "in removed data" and "not in removed data" HumanEval subsets, indicating that the contamination mechanism is not memorization but rather a general boost to the model's performance on HumanEval-style problems.67% similarUnverifiedThe released checkpoint was trained with a chunk size of 16; changing the sampling chunk size to 64 causes HumanEval to drop from 76% to approximately 20%, and GSM8K to drop from a high position to 2.2%.65% similarUnverified基准污染（Benchmark Contamination）导致HumanEval、MMLU等主流基准区分度下降，部分模型HumanEval得分超过95%、MMLU超过90%但实际能力平庸63% similarUnverified2024年多项研究发现，部分模型在去污染版本的评测集上得分下降幅度可达10-20个百分点63% similarUnverifiedThe developer built the ishuman benchmark, in which 2.0 was judged to be human 23.5% of the time, the base model abliterated 0.3%, the official original version 15.1%, and the base model with prompting 6.8%.61% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1013060API
curl https://kongchang.com/api/v1/knowledge/claims/1013060MCP
get_claim(id=1013060)