Verified65% confidenceFactExact time
Training data contamination is the primary challenge faced by AI programming benchmarks, as test cases come from public GitHub repositories, which are the same sources used as pretraining data for large models.
3
Sources
65%
Confidence
Long-term
Relevance
7/10/2026
First Seen
Sources
OpenAI Abandons SWE-Bench Pro: The Trust Crisis in AI Coding Benchmarks and What Comes Next
hackernewshackernews7/8/2026
Related Claims
UnverifiedEvaluations based on open-source codebases suffer from data contamination issues, as these codebases may have been included in large model training data.79% similarVerifiedThe training data sources for specialized programming models include open-source code from GitHub, competitive programming solutions, technical documentation, and code review records, and the models are optimized for benchmarks such as HumanEval, MBPP, and SWE-bench.77% similarUnverified数据污染问题源于LLM预训练语料来自Common Crawl等大规模网络爬取，可能包含公开的基准测试题目及答案75% similarUnverifiedEvaluating AI's capacity for scientific discovery requires setting up truly held-out tests, benchmarking against conventional automated methods, and honestly confronting training data contamination and prior knowledge interference.74% similarUnverifiedBenchmark optimization suffers from training data contamination, with many models boosting their scores by having 'seen' the test questions before.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/465779API
curl https://kongchang.com/api/v1/knowledge/claims/465779MCP
get_claim(id=465779)