Masked Diffusion Model
A specific form of discrete diffusion model in which the noising process randomly replaces tokens with a [MASK] token, and the reverse process learns to predict the original masked tokens.
Timeline (last 90 days)
Anari et al.'s scheme requires one score query for every token unmasked, so the number of queries grows linearly with the block size.
The noising process of a masked diffusion model (MDM) progressively replaces tokens in the sequence with a special [MASK] token, and the training objective is for the model to learn to predict the original token at each masked position based on the currently partially visible sequence.
During MDM generation, the conditional distribution at each position can be computed simultaneously, which naturally supports parallel decoding, in sharp contrast to the way autoregressive models generate tokens one by one.
Anari et al. (2026) proposed using rejection sampling to correct the distributional bias in parallel decoding of masked diffusion models, decomposing the log acceptance probability into a sum of token-level terms.
SIRS (Single-Increment Rejection Sampler) randomly samples one increment and uses a single query to estimate the log acceptance probability of the entire block, thereby reducing query cost while preserving unbiasedness.
Masked diffusion models can decode in parallel, support generation in arbitrary order, and are naturally suited to handling discrete representations of multiple modalities.
The latest research proposes the first trimodal masked diffusion model pretrained from scratch, trained simultaneously on text, image-text, and audio-text data.
All Facts (7)
Anari et al.'s scheme requires one score query for every token unmasked, so the number of queries grows linearly with the block size.
50%UnverifiedThe noising process of a masked diffusion model (MDM) progressively replaces tokens in the sequence with a special [MASK] token, and the training objective is for the model to learn to predict the original token at each masked position based on the currently partially visible sequence.
50%UnverifiedDuring MDM generation, the conditional distribution at each position can be computed simultaneously, which naturally supports parallel decoding, in sharp contrast to the way autoregressive models generate tokens one by one.
50%UnverifiedAnari et al. (2026) proposed using rejection sampling to correct the distributional bias in parallel decoding of masked diffusion models, decomposing the log acceptance probability into a sum of token-level terms.
50%UnverifiedSIRS (Single-Increment Rejection Sampler) randomly samples one increment and uses a single query to estimate the log acceptance probability of the entire block, thereby reducing query cost while preserving unbiasedness.
50%UnverifiedMasked diffusion models can decode in parallel, support generation in arbitrary order, and are naturally suited to handling discrete representations of multiple modalities.
50%UnverifiedThe latest research proposes the first trimodal masked diffusion model pretrained from scratch, trained simultaneously on text, image-text, and audio-text data.
50%