Verified65% confidenceFactExact time
The three major benchmarks MMLU, GSM8K, and HumanEval have all experienced varying degrees of contamination, with researchers finding content in the training data of multiple mainstream models that overlaps highly with test questions
3
Sources
65%
Confidence
Long-term
Relevance
7/10/2026
First Seen
Sources
LLM Security Benchmarking: Current State, Challenges, and Practical Guide
hackernewshackernews7/6/2026
Related Claims
UnverifiedTraditional benchmarks such as MMLU and GSM8K focus on academic ability, with fixed questions that are easily covered by training data, leading to data contamination problems.77% similarUnverifiedMainstream evaluation sets such as MMLU, HumanEval, and GSM8K have long been publicly available on the internet, making it highly likely that models have been exposed to these questions during pretraining or fine-tuning.76% similarUnverifiedMultiple studies in 2024 found that the strong performance of certain models on standard benchmarks resulted from benchmark test questions being mixed into the training data (data contamination).75% similarVerifiedWhen test questions may have appeared in training data, high scores do not truly represent generalization ability — this is known as benchmark contamination.70% similarUnverifiedThe core assumption of the research is that shared training data and similar architectures cause more capable LLMs to exhibit higher behavioral consistency with one another, thereby generating systemic risk68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/461660API
curl https://kongchang.com/api/v1/knowledge/claims/461660MCP
get_claim(id=461660)