[KongchangAI]
· 2 min read· 1,363 words

Farewell to Independent Test Sets: How Evaluability Theory Is Reshaping LLM Evaluation

Farewell to Independent Test Sets: How Evaluability Theory Is Reshaping LLM Evaluation

In the foundation model era, the test set independence assumption fails; the paper proposes a formal theoretical framework for evaluability.

The article introduces the arXiv paper *Toward a Theory of Evaluability*, noting that heterogeneous foundation model training data makes test set leakage nearly inevitable and undermines the independence assumption underlying traditional evaluation. The paper seeks a formal definition of evaluability that does not rely on independent test sets, framing evaluation as inference under unknown distribution shift. Safety, calibration, and fairness evaluations are all affected by data contamination, and the credibility of results hinges on test-training data independence. Practically, contamination should be quantified via membership inference and data provenance, independence assumption status should be explicitly reported, and training-evaluation pipeline co-design should be advanced.

When Test Sets Are No Longer "Clean": The Foundation of LLM Evaluation Is Shaking

Machine learning model evaluation has long rested on a classic assumption: test sets are strictly isolated during training. However, as model scale balloons to the level of foundation models, this assumption becomes increasingly untenable. According to the arXiv paper Toward a Theory of Evaluability, the training data sources of large foundation models are vast and heterogeneous, making test set leakage almost inevitable—and the legitimacy of traditional evaluation methods is being eroded.

More complicating is that the evaluation dimensions of modern models are no longer limited to task accuracy. Safety, calibration, fairness, and other multi-axis metrics are required simultaneously, and each dimension may face the risk of data contamination. The authors therefore raise a fundamental question: when test set independence cannot be guaranteed, how else can we define "evaluability"?

rss source: Toward a Theory of Evaluability

Test Set Leakage refers to evaluation test samples appearing directly or indirectly in a model's training data, causing the model to "recall" rather than genuinely "generalize" at test time. In the foundation model setting, leakage pathways are diverse: pretraining corpora may have crawled public evaluation sets or their approximate paraphrased versions; instruction-tuning data may have been mixed with standard answers to test questions; and even during training, models may indirectly encounter test distributions through data augmentation, synthetic data generation, and similar stages. The consequence of leakage is that evaluation metrics are systematically overestimated, and this bias is difficult to detect through simple checks because the model does not "memorize verbatim" but absorbs information in a semantically approximate manner. Quantifying leakage typically requires membership inference, data provenance, or likelihood-based detection methods, but these methods themselves have limitations.

Formalizing Evaluability: From Assumption to Theory

This work is not content merely to point out the problem; it attempts to establish a formal definition of "evaluability" that does not rely on the independent test set assumption. The core idea is to treat evaluation as an inference problem under unknown shifts in the data distribution, rather than as a simple held-out set performance measurement.

The paper asks: what combinations of model, data, and evaluation protocol can still yield meaningful conclusions without a clean test set? This involves re-examining the relationships among evaluation objectives, model access privileges (e.g., whether querying a model's internal state is permitted), and the data generation process. Although the abstract does not provide a complete technical solution, the proposal of this theoretical framework itself already points to a paradigm shift in evaluation—from "measuring performance" to "arguing credibility."

Membership Inference is a class of techniques for determining whether a given sample was used to train a model. The basic idea is that a model's prediction confidence or loss value for training samples is often systematically lower than for unseen samples; by training a binary classifier or designing statistical tests, one can estimate the probability that a sample was "seen." In evaluation theory, membership inference is used to quantify the degree of test set leakage: if a large fraction of test samples are confidently judged to be training members, then the independence assumption of that test set does not hold. However, membership inference has limited effectiveness on foundation models because models generally have high confidence on in-distribution samples, reducing discriminability; moreover, attackers or evaluators usually have only black-box access to the model, further increasing inference difficulty. Nevertheless, it remains one of the few operational leakage detection methods currently available.

The Challenge of Multi-Axis Evaluation: Safety, Calibration, and Fairness

Foundation model evaluation typically focuses on multiple attributes simultaneously. Take safety as an example: the traditional approach is to compute a violation rate on a labeled safety test set. But if samples from that test set appeared in the pretraining corpus, the model's performance will be overestimated. Calibration evaluation is equally fragile: a model's estimate of its own uncertainty may become "overconfident" or "overly conservative" because it has seen the test questions.

The fairness dimension is even trickier because fairness testing often relies on specific demographic subsets, and data for these subsets is more likely to be inadvertently covered during training. The paper emphasizes that these multi-axis evaluation problems share the same structure: the credibility of evaluation results depends on the degree of independence between test data and training data, and in the foundation model era this independence can no longer be assumed to be true by default.

Calibration measures the consistency between a model's output confidence and its actual accuracy: if a model gives 80% confidence to all its predictions, then approximately 80% of those predictions should be correct. Good calibration is crucial for decision-making in high-risk settings because downstream systems need to rely on confidence to set thresholds or trigger human review. Calibration evaluation typically uses metrics such as Expected Calibration Error (ECE), computing the deviation between confidence bins and actual accuracy on a held-out test set. However, when test samples have been seen by the model during training, the model often exhibits unusually high confidence in them along with high accuracy, causing calibration error to be underestimated and presenting "falsely good calibration." More insidiously, leakage may affect only specific subpopulations, causing local calibration distortion while overall metrics appear normal.

Implications for Evaluation Practice

Although the paper leans theoretical, its implications for practice are clear. Evaluators need to develop new protocols to quantify the impact of data contamination—for example, using membership inference or data provenance to estimate the probability that a test sample has been "seen" by the model. At the same time, evaluation reports should explicitly state the degree to which the independence assumption is satisfied, rather than simply presenting measured numbers.

Another implication is that evaluability theory may promote "evaluation-aware" model development—consciously reserving certain data regions during training as verifiable evaluation anchors. This requires tighter co-design between training pipelines and evaluation pipelines, rather than the currently common linear process of training first and evaluating afterward.

Data Provenance aims to track the specific sources, flow paths, and usage records of training data, providing a basis for determining whether test samples overlap with training data. In foundation model evaluation, data provenance can take various forms: recording the crawl sources and timestamps of pretraining corpora, deduplicating and source-labeling instruction-tuning data, and maintaining "lineage" information for test sets. Ideally, evaluators can estimate overlap rates by comparing hash fingerprints or semantic fingerprints of test sets against training data sources. But the practical challenges are enormous: the training data of many foundation models is not fully public, and provenance chains are easily broken during data cleaning, mixing, and synthesis. Therefore, data provenance is more of auxiliary evidence and must be combined with methods such as membership inference to provide a relatively reliable judgment of the degree to which the independence assumption is satisfied.

Open Questions and Future Directions

The paper concludes with open questions: Is evaluability a quantifiable property? Do different evaluation objectives (e.g., safety vs. accuracy) differ in their sensitivity to data independence? As model capabilities continue to improve, evaluation theory itself needs to iterate. The value of this work lies in elevating "test set leakage" from an engineering nuisance to a fundamental problem requiring a theoretical response, setting a clear agenda for future research.

One-line Summary

In the foundation model era, the test set independence assumption fails; the paper proposes a formal theoretical framework for evaluability.

Paragraph Summary

The article introduces the arXiv paper Toward a Theory of Evaluability, pointing out that the vast and heterogeneous training data of foundation models makes test set leakage almost inevitable, shaking the independence assumption on which traditional evaluation relies. The paper attempts to establish a formal definition of "evaluability" that does not depend on the independent test set assumption, treating evaluation as an inference problem under unknown shifts in the data distribution. Multi-axis evaluations such as safety, calibration, and fairness are all affected by data contamination, and the credibility of evaluation results depends on the degree of independence between test and training data. In practice, contamination must be quantified through membership inference and data provenance, the satisfaction status of the independence assumption must be explicitly reported, and co-design of training and evaluation pipelines must be promoted.

#evaluability theory#test set leakage#foundation model evaluation#membership inference#data provenance#calibration#fairness evaluation#LLM benchmarking
Share:

Related articles