Benchmark Contamination
A phenomenon in which the questions or answers of a public benchmark enter a model's training corpus, so that benchmark performance no longer reflects genuine generalization ability. It is a key methodological issue in LLM evaluation.
Timeline (last 90 days)
The probability that public evaluation sets appear in training data changes with model updates, so scores on old evaluations lose comparability; API models undergo silent updates, and the same model name may correspond to different weights at different times.
Benchmark contamination means that the questions or answers of a public evaluation set have entered the model's training corpus in some form, so that the model's performance on the evaluation cannot reflect its true generalization ability; as training data scale grows, the time window from an evaluation question's release to its being memorized becomes ever shorter.
The rigorous way to address benchmark contamination is to distinguish 'closed-book' from 'open-book' settings, or to use evaluation sets released later than the model's training cutoff (contamination-free benchmark).
All Facts (3)
The probability that public evaluation sets appear in training data changes with model updates, so scores on old evaluations lose comparability; API models undergo silent updates, and the same model name may correspond to different weights at different times.
50%UnverifiedBenchmark contamination means that the questions or answers of a public evaluation set have entered the model's training corpus in some form, so that the model's performance on the evaluation cannot reflect its true generalization ability; as training data scale grows, the time window from an evaluation question's release to its being memorized becomes ever shorter.
50%UnverifiedThe rigorous way to address benchmark contamination is to distinguish 'closed-book' from 'open-book' settings, or to use evaluation sets released later than the model's training cutoff (contamination-free benchmark).
50%