Corpus Sampling Divergence
A statistical measure in machine learning that quantifies the degree of output disagreement between two predictive models trained on separate corpora independently sampled from the same population. It is used to evaluate a model's sensitivity to the randomness inherent in the data sampling process.
Timeline (last 90 days)
The study quantifies corpus sampling divergence for a masked text reconstruction task, defined as the difference in predictor outputs when two corpora are independently collected from the same population and each is used to train a predictor.
According to the classical 1/n scaling law in statistics, quadrupling the sample size should reduce divergence by approximately 75%, but in practice divergence decreased by only 5% to 40%.
The slow decline in divergence is traced to the smoothing of rare contexts, as increasing the amount of data cannot effectively reduce sampling variance in sparse regions.
Under the long-tail distribution of language tasks, the uncertainty-reduction gains from adding data may be far below what the 1/n intuition would suggest.
All Facts (4)
The study quantifies corpus sampling divergence for a masked text reconstruction task, defined as the difference in predictor outputs when two corpora are independently collected from the same population and each is used to train a predictor.
50%UnverifiedAccording to the classical 1/n scaling law in statistics, quadrupling the sample size should reduce divergence by approximately 75%, but in practice divergence decreased by only 5% to 40%.
50%UnverifiedThe slow decline in divergence is traced to the smoothing of rare contexts, as increasing the amount of data cannot effectively reduce sampling variance in sparse regions.
50%UnverifiedUnder the long-tail distribution of language tasks, the uncertainty-reduction gains from adding data may be far below what the 1/n intuition would suggest.
50%