benchmark contamination
A phenomenon in large language model evaluation in which benchmark questions or answers appear in a model's training corpus, causing inflated scores on those benchmarks that do not accurately reflect the model's true generalization ability. It represents a central challenge to the credibility and validity of LLM evaluation practices.
Core Facts
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
Benchmark Contamination is a significant challenge in the current field of large model evaluation, where publicly available test questions may already be included in model training data, causing evaluation results to be distorted.
When test questions may have appeared in training data, high scores do not truly represent generalization ability — this is known as benchmark contamination.
Timeline (last 90 days)
Benchmark leakage refers to test questions or their variants appearing in a model's pre-training corpus during the pre-training phase.
Benchmark saturation refers to a situation where the questions or solutions in a test set have been covered by a model's training data, causing high scores to no longer represent true capability.
Since the training corpora of modern LLMs are typically collected through large-scale internet crawls and many benchmark test questions have long been publicly available online, benchmark data leakage is almost unavoidable.
There is a fundamental tension between the requirements of transparent and reproducible science and the fact that transparency itself can undermine measurement validity.
Science demands transparency and reproducibility, but benchmark transparency itself undermines the validity of measurement, creating a fundamental tension between the two.
Recent research indicates that some models perform 10–15 percentage points higher on known contaminated benchmark subsets than on uncontaminated subsets.
Some vendors tend to over-optimize for evaluation sets (benchmark gaming/benchmark contamination), leading to inflated scores but insufficient generalization ability.
When test questions may have appeared in training data, high scores do not truly represent generalization ability — this is known as benchmark contamination.
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
使用模型训练过程中未接触过的unseen样本可以有效避免评测污染（benchmark contamination），提高测试可信度
All Facts (11)
Benchmark contamination refers to the phenomenon where test questions may have already appeared in training data, causing high scores to not truly represent generalization ability.
85%VerifiedBenchmark Contamination is a significant challenge in the current field of large model evaluation, where publicly available test questions may already be included in model training data, causing evaluation results to be distorted.
75%VerifiedWhen test questions may have appeared in training data, high scores do not truly represent generalization ability — this is known as benchmark contamination.
65%UnverifiedBenchmark leakage refers to test questions or their variants appearing in a model's pre-training corpus during the pre-training phase.
50%UnverifiedBenchmark saturation refers to a situation where the questions or solutions in a test set have been covered by a model's training data, causing high scores to no longer represent true capability.
50%UnverifiedSince the training corpora of modern LLMs are typically collected through large-scale internet crawls and many benchmark test questions have long been publicly available online, benchmark data leakage is almost unavoidable.
50%UnverifiedThere is a fundamental tension between the requirements of transparent and reproducible science and the fact that transparency itself can undermine measurement validity.
50%UnverifiedScience demands transparency and reproducibility, but benchmark transparency itself undermines the validity of measurement, creating a fundamental tension between the two.
50%UnverifiedRecent research indicates that some models perform 10–15 percentage points higher on known contaminated benchmark subsets than on uncontaminated subsets.
50%UnverifiedSome vendors tend to over-optimize for evaluation sets (benchmark gaming/benchmark contamination), leading to inflated scores but insufficient generalization ability.
50%Unverified使用模型训练过程中未接触过的unseen样本可以有效避免评测污染（benchmark contamination），提高测试可信度
50%