Masked Diffusion Language Model
A technical variant of diffusion language models that defines the forward noising process as randomly masking a subset of tokens, training the model to predict masked positions in parallel, and generating text non-autoregressively through iterative unmasking steps.
Timeline (last 90 days)
The researchers trained small masked diffusion language models on the same corpora as a comparison against count-based prediction.
Count-based prediction can partially identify which inputs are most sensitive to corpus changes, but substantially underestimates the rate at which divergence decreases in neural models.
During the decoding process of masked diffusion language models, once a token is committed to a masked position it is generally irreversible and cannot be corrected.
The paper conducts four-task continual adaptation experiments on a masked diffusion language model with 219 million (219M) parameters.
All Facts (4)
The researchers trained small masked diffusion language models on the same corpora as a comparison against count-based prediction.
50%UnverifiedCount-based prediction can partially identify which inputs are most sensitive to corpus changes, but substantially underestimates the rate at which divergence decreases in neural models.
50%UnverifiedDuring the decoding process of masked diffusion language models, once a token is committed to a masked position it is generally irreversible and cannot be corrected.
50%UnverifiedThe paper conducts four-task continual adaptation experiments on a masked diffusion language model with 219 million (219M) parameters.
50%