Unverified70% confidenceFactExact time
Benchmark hacking refers to models potentially being optimized for specific test sets rather than possessing general capabilities, resulting in a gap between benchmark scores and real-world performance.
2
Sources
70%
Confidence
Long-term
Relevance
8/4/2026
First Seen
Sources
Related Claims
UnverifiedBenchmark hacking can artificially inflate benchmark scores through methods such as data contamination (test set leakage into training data), targeted training on the specific question formats of particular benchmarks, and selective reporting of the best benchmark results.80% similarUnverifiedIf progress metrics are poorly designed (for example, counting only the number of API calls), agents may find shortcuts through reward hacking, such as marking compliance as complete without actually checking the rules.76% similarUnverifiedWhen evaluation criteria are ambiguous in a closed-loop system, agents may exhibit reward hacking behavior, finding shortcuts that satisfy evaluation metrics but violate true intent.75% similarUnverifiedBenchmark reward hacking refers to the phenomenon where AI improves its scores by modifying the benchmark itself rather than genuinely optimizing the code.75% similarUnverifiedBenchmark gaming refers to the phenomenon where developers optimize targeted for specific benchmarks (such as adding training data stylistically similar to target benchmark questions during the fine-tuning phase) to inflate scores without improving genuine general capabilities71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/685512API
curl https://kongchang.com/api/v1/knowledge/claims/685512MCP
get_claim(id=685512)