How Corpus Sampling Affects Masked Text Reconstruction: A Quantitative Study of Predictor Disagreement

An Overlooked Question: How Much Does Model Output Change When You Resample the Training Data?

Machine learning practice often relies on an implicit assumption: if you resample a new batch of training data from the same population, the model's predictions should remain roughly the same. But what exactly does "roughly the same" mean? This study systematically quantifies corpus-sampling disagreement for the masked-text reconstruction task — that is, how much the outputs of two predictors trained on independently sampled corpora from the same population will diverge.

The researchers built a smoothed predictor based on neighboring token-pair counts and evaluated it on GPT-2's full 50,257-token vocabulary. A key modeling choice was treating each source document as a single observation unit, rather than treating each token as an independent sample. This better reflects the actual data-generating process, since tokens within the same document are highly correlated.

Masked-text reconstruction is a classic self-supervised task in natural language processing: given a passage of text, randomly mask some tokens and require the model to predict what was hidden. This mechanism — known as Masked Language Modeling (MLM) — is the core of BERT-style pretraining. In this paper's framework, the predictor takes neighboring token-pair context as input and outputs a probability distribution over the masked token. This task was chosen as the research target because it is simple enough to construct closed-form count-based predictors, yet close enough to real LLM pretraining scenarios to make the findings transferable.

Core Contribution: A Disagreement Prediction Formula Computable in Advance

The most valuable contribution of this work is the derivation of a first-order formula for predicting the expected disagreement between predictors fitted on two independent corpora. This formula has several noteworthy properties:

• It accounts for three realistic factors: variable document length, random context counts, and correlations across token offsets;
• More importantly, it can be computed using only a single independent reference corpus, with no access to any target training corpus.

In other words, the researchers provide a way to estimate "how much uncertainty would a new batch of data introduce" without ever touching the actual training data — a practically useful tool for evaluating model robustness and data dependency.

The "first-order formula" is derived from the statistical delta method: performing a first-order Taylor expansion of a nonlinear function around its expected value, approximating the complex random quantity as a linear combination to obtain an analytic expression for its variance. In this setting, predictor outputs are nonlinear functions of corpus counts; through the delta method, the expected disagreement between two predictors trained on independent corpora is decomposed into a sum of statistics that can be estimated directly from a single reference corpus. The key challenge in this derivation is that unequal document lengths and within-sequence token correlations invalidate the naive i.i.d. assumption. The authors address both complications by treating documents as the fundamental observation unit and introducing covariance terms across token offsets.

Experimental Findings: Disagreement Decreases Far More Slowly Than Expected

The study validates its approach on three datasets: TinyStories, WikiText-103, and CNN/DailyMail. The experimental setup scales each corpus from 512 chunks to 2,048 chunks (each chunk containing 64 tokens).

Under the classical $1/n$ scaling rule, quadrupling the sample size should reduce disagreement by roughly 75%. The actual results were surprising: disagreement decreased by only 5% to 40%, far below theoretical expectations.

The researchers trace this slow decline to smoothing of rarely seen contexts. Because many contexts appear extremely infrequently in the corpus, adding more data does not effectively reduce sampling variance in these sparse regions, which slows the overall convergence of disagreement. Their prediction formula accurately tracks this slow-decay trend, validating the method's effectiveness.

Key to Prediction Accuracy: Reference Corpus Size Matters More Than Correlation Modeling

One counterintuitive but important finding is that prediction accuracy depends more on the size of the reference corpus than on how correlations are modeled.

Specifically, when the reference corpus is too small, it systematically underestimates true disagreement. To address this, the researchers introduce a source-split jackknife that uses only reference data, reducing this calibration error by 57% to 59%.

After this correction, different approximation methods suit different scenarios:

• When the reference corpus is smaller than the target corpus, the simpler smoothed-multinomial approximation is most accurate;
• When the reference corpus is larger than the target corpus, the nonlinear source bootstrap performs best.

This provides clear practical guidance for method selection.

The jackknife is a classic nonparametric resampling technique for estimating the bias and variance of a statistic. The basic idea is to iteratively remove one observation from the dataset, recompute the statistic, and then use these leave-one-out estimates to correct the systematic bias of the original estimate. The "source-split jackknife" proposed here is a variant of the standard jackknife: since the paper uses documents as the unit of observation, each deletion removes an entire document rather than a single token, and the operation is applied only to the reference corpus (not the target training corpus), enabling bias correction without accessing the target data. This approach is especially effective when the reference corpus is small and variance terms are systematically underestimated — a critical engineering component of the paper's methodological pipeline.

Extension to Neural Models: The Limits of Count-Based Methods

The study concludes by comparing count-based prediction against actual neural networks. Small masked diffusion language models were trained on the same corpora, revealing a two-sided picture of count-based methods:

On one hand, count-based prediction can partially identify which inputs are most sensitive to corpus changes — suggesting the method captures some essential patterns of data dependency.

On the other hand, it significantly underestimates how quickly neural model disagreement decreases with more data. Neural models converge much faster than count-based predictors as data grows. This implies that neural networks possess generalization and information-sharing mechanisms that count models lack, allowing them to more effectively leverage data even in sparse contexts.

Masked diffusion language models are a class of generative models that have emerged in recent years, bringing the ideas of diffusion processes into the discrete text domain: during training, text is progressively corrupted with masking noise, and during inference, the complete text is gradually restored through denoising steps. Unlike BERT's single-step MLM pretraining, these models iterate multiple steps during inference and can theoretically model more complex joint distributions over tokens. This paper uses them to compare against count-based predictors both because their training objective directly corresponds to the masked reconstruction task, and to probe the fundamental difference in data utilization efficiency between parameterized neural networks and count-based methods. Experimental results show that neural models generalize far better than count predictors on sparse contexts — the mechanism behind this gap remains an open question.

Implications for Practitioners

Although this study is primarily theoretical, it offers valuable insights into the relationship between data and model behavior:

First, do not naively apply $1/n$ intuition to estimate "how much uncertainty more data will eliminate" — in language tasks with long-tail distributions, the gains may be far smaller than expected.

Second, when assessing a model's sensitivity to data, you can use an independent reference corpus to forecast disagreement without repeatedly resampling and retraining.

Third, the systematic gap between count-based methods and neural models is a reminder to be cautious when using simple proxy models to reason about the behavior of large models.