Unverified50% confidenceFactExact time
Around 2023, multiple academic benchmarks including BIG-bench and HELM were pointed out to have test set contamination issues, meaning the models' training data may have already included evaluation questions, leading to inflated scores.
1
Sources
50%
Confidence
Long-term
Relevance
9/25/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.79% similarVerifiedBenchmark Contamination is a significant challenge in the current field of large model evaluation, where publicly available test questions may already be included in model training data, causing evaluation results to be distorted.78% similarUnverifiedIndustry benchmarks are shifting from MMLU and HumanEval to next-generation benchmarks closer to real-world engineering scenarios such as SWE-bench, because the former carry data contamination risks and lack sufficient difficulty to differentiate top models.73% similarVerified若用含测试集的全量数据计算均值和标准差，会导致预处理泄漏，使评估指标虚高，无法真实反映模型泛化能力71% similarUnverified传统回归测试面临核心困境：全量回归耗时过长，而随机选择可能遗漏关键场景71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/949534API
curl https://kongchang.com/api/v1/knowledge/claims/949534MCP
get_claim(id=949534)