Masked Language Modeling
A classical self-supervised pretraining task in natural language processing. Given a text sequence, a subset of tokens is randomly masked and the model is trained to predict the masked content from the surrounding context. This mechanism forms the core pretraining objective of BERT-family models.
Timeline (last 90 days)
The study quantifies corpus sampling divergence for a masked text reconstruction task, defined as the difference in predictor outputs when two corpora are independently collected from the same population and each is used to train a predictor.
The study treats each source document as a unit of observation, rather than treating each token as an independent sample.
All Facts (2)
The study quantifies corpus sampling divergence for a masked text reconstruction task, defined as the difference in predictor outputs when two corpora are independently collected from the same population and each is used to train a predictor.
50%UnverifiedThe study treats each source document as a unit of observation, rather than treating each token as an independent sample.
50%