Data Contamination

Data contamination refers to test problems having already appeared in pretraining corpora, causing a model's high scores to more closely reflect memorization rather than reasoning.

Data contamination refers to a situation where a model has already seen data from the test set during training, causing benchmark scores to be inflated.

The training corpora of large language models may include unpublished academic preprints and research notes, raising concerns about data contamination.

Data leakage falls into two categories—target leakage and train-test contamination—and leads to systematic overestimation of model performance.

Training set contamination is the core controversy in the current AI benchmark evaluation field; models may answer questions through memorization rather than reasoning, leading to inflated results.

Because extensive discussions of benchmark questions exist on the internet, models may have already seen the answers during training, causing test scores to be inflated—a phenomenon known as the 'contamination problem'.

The difficulty of building evaluation sets lies in covering edge cases, avoiding data contamination, and ensuring alignment between evaluation criteria and real business value.

Some researchers have attempted to mitigate data contamination by constructing "future datasets" or dynamically generating new questions, but such methods have not yet become an industry standard.

A perfect score on a benchmark may stem from three possible causes: a capability breakthrough, data contamination, or changes in evaluation configuration or version.

Using unpublished papers reduces the risk of data contamination from the model having seen the answers in its training data.

Evaluations based on open-source codebases suffer from data contamination issues, as these codebases may have been included in large model training data.

The degree of harm caused by data contamination is positively correlated with a model's memorization capacity; small models with limited capacity cannot benefit from contaminated data even when exposed to it.

The data contamination problem refers to the possibility that a model may have already encountered ideas or partial proofs during training that researchers have not yet formally published.

If the model memorized the answers to contaminated benchmarks during pre-training, information loss from compressing the context will not affect performance, rendering the reported 'lossless compression' conclusion meaningless.

当模型训练数据包含评测集题目或高度相似内容时，模型可能通过记忆而非真正理解获得高分