Benchmark Contamination
A phenomenon in which an AI model has been exposed to evaluation benchmark questions during the training phase, causing inflated benchmark scores that do not reflect the model's actual generalization capability. It represents a significant challenge in the field of AI evaluation.
Core Facts
Benchmark contamination refers to a model having already seen test questions during training, resulting in inflated scores that fail to reflect true generalization ability.
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
Benchmark data contamination refers to the phenomenon where, after a test set is made public, its questions may appear in the pre-training corpora of subsequent models, causing scores to reflect memorization rather than genuine reasoning ability.
Timeline (last 90 days)
Benchmark contamination refers to the possibility that after an evaluation set is made public, models may be trained specifically for it, so benchmarks need to be continuously updated or keep a private held-out set to maintain credibility
Benchmark contamination refers to the situation where benchmark questions have entered a model's training data, causing the model to "recall answers" rather than genuinely reason, inflating benchmark scores and failing to reflect actual capability.
Common measures to mitigate benchmark contamination include using dynamically generated new questions, keeping the question bank confidential before release, or mixing evaluations across multiple dimensions to reduce the weight of any single dataset.
A study published on arXiv (arXiv:2609.02899) proposes that concerns about benchmark contamination should be distinguished into two separate questions: whether absolute scores are inflated and whether the ranking order of models is changed.
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
All Facts (7)
Benchmark contamination refers to a model having already seen test questions during training, resulting in inflated scores that fail to reflect true generalization ability.
90%VerifiedBenchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedBenchmark data contamination refers to the phenomenon where, after a test set is made public, its questions may appear in the pre-training corpora of subsequent models, causing scores to reflect memorization rather than genuine reasoning ability.
80%UnverifiedBenchmark contamination refers to the possibility that after an evaluation set is made public, models may be trained specifically for it, so benchmarks need to be continuously updated or keep a private held-out set to maintain credibility
50%UnverifiedBenchmark contamination refers to the situation where benchmark questions have entered a model's training data, causing the model to "recall answers" rather than genuinely reason, inflating benchmark scores and failing to reflect actual capability.
50%UnverifiedCommon measures to mitigate benchmark contamination include using dynamically generated new questions, keeping the question bank confidential before release, or mixing evaluations across multiple dimensions to reduce the weight of any single dataset.
50%UnverifiedA study published on arXiv (arXiv:2609.02899) proposes that concerns about benchmark contamination should be distinguished into two separate questions: whether absolute scores are inflated and whether the ranking order of models is changed.
50%