MDLM
MDLM (Masked Diffusion Language Model) is a representative work in discrete masked diffusion language modeling that applies the stepwise denoising framework of diffusion models to discrete token spaces for language generation.
Core Facts
Timeline (last 90 days)
Mainstream samplers such as MDLM and SEDD rely on confidence scores from the current timestep to make greedy decisions when selecting fixed positions, and once a position is decoded it no longer participates in subsequent denoising cycles.
Unlike autoregressive models that strictly generate from left to right, MDLMs theoretically allow parallel filling of arbitrary positions.
Masked Diffusion Language Modeling (MDLM) operates on discrete token spaces, where the forward process randomly replaces tokens with [MASK] and the reverse process trains the model to progressively restore the masked tokens.
MDLM and SEDD are representative works in discrete diffusion models.
Protein generation models such as ESM3 and MDLM belong to the discrete diffusion model architecture.
Early diffusion language model work (such as MDLM and PLAID) mainly made progress on the perplexity metric, but still showed significant gaps in open-ended generation and complex reasoning tasks.
In arena evaluations on continuation benchmarks, Reviser significantly outperforms two discrete diffusion/masked language model baselines, SEDD and MDLM.
Discrete masked diffusion draws on BERT's masked language modeling idea—randomly masking and restoring sequences during training and iteratively filling in from a fully masked sequence at inference—with representative works including MDLM and SEDD.
Typical methods for discrete diffusion include D3PM (adding noise via a token transition matrix) and MDLM (borrowing BERT's masking strategy to replace tokens with [MASK]).
Diffusion-LM is widely recognized as the pioneering work in the direction of diffusion language models.
4 more timeline events
All Facts (16)
扩散模型的核心原理分为前向扩散过程（逐步添加高斯噪声）和反向去噪过程（从噪声中还原图像）两个阶段
90%Unverified早期的离散扩散语言模型如 MDLM、SEDD 通过在 Token 嵌入空间或掩码空间中定义扩散过程来解决文本离散性挑战
85%UnverifiedDiffusion-LM is widely recognized as the pioneering work in the direction of diffusion language models.
70%UnverifiedMainstream samplers such as MDLM and SEDD rely on confidence scores from the current timestep to make greedy decisions when selecting fixed positions, and once a position is decoded it no longer participates in subsequent denoising cycles.
50%UnverifiedUnlike autoregressive models that strictly generate from left to right, MDLMs theoretically allow parallel filling of arbitrary positions.
50%UnverifiedMasked Diffusion Language Modeling (MDLM) operates on discrete token spaces, where the forward process randomly replaces tokens with [MASK] and the reverse process trains the model to progressively restore the masked tokens.
50%UnverifiedMDLM and SEDD are representative works in discrete diffusion models.
50%UnverifiedProtein generation models such as ESM3 and MDLM belong to the discrete diffusion model architecture.
50%UnverifiedEarly diffusion language model work (such as MDLM and PLAID) mainly made progress on the perplexity metric, but still showed significant gaps in open-ended generation and complex reasoning tasks.
50%UnverifiedIn arena evaluations on continuation benchmarks, Reviser significantly outperforms two discrete diffusion/masked language model baselines, SEDD and MDLM.
50%UnverifiedDiscrete masked diffusion draws on BERT's masked language modeling idea—randomly masking and restoring sequences during training and iteratively filling in from a fully masked sequence at inference—with representative works including MDLM and SEDD.
50%UnverifiedTypical methods for discrete diffusion include D3PM (adding noise via a token transition matrix) and MDLM (borrowing BERT's masking strategy to replace tokens with [MASK]).
50%UnverifiedBuilding a minimum viable diffusion language model can adopt the discrete mask diffusion approach, referencing the MDLM or D3PM framework, where the forward process replaces tokens with [MASK] at timestep-dependent probabilities via a transition matrix.
50%Unverified离散扩散模型发展经历D3PM（2021年Austin等人）、MDLM（2023年Sahoo等人）、SEDD（2023年Lou等人）等里程碑，但模型规模通常不超过1B参数
50%UnverifiedMDLM（Sahoo et al., 2024）专注于吸收态扩散，前向过程只将 token 替换为 [MASK]
50%Unverified离散扩散语言模型主流方案包括：将token映射到连续嵌入空间进行扩散（如Diffusion-LM），或设计离散扩散过程用掩码模拟加噪（如D3PM、MDLM）
50%