Discrete Diffusion Model
A class of generative models that applies the diffusion framework to discrete token sequences. The forward process progressively masks or replaces tokens, while the reverse process reconstructs the original sequence. These models support parallel writing to multiple positions and are commonly used for text, speech, code, and protein sequence generation.
Timeline (last 90 days)
MDLM and SEDD are representative works in discrete diffusion models.
COFFEE is a plug-and-play framework that adds sequence-level goal guidance to discrete diffusion models without retraining the pretrained model.
Discrete diffusion models construct complete sequences by iteratively and in parallel resolving multiple tokens across multiple steps, unlike the left-to-right word-by-word generation of autoregressive models.
When guiding discrete diffusion models with sequence-level goals, enumerating all possible completion combinations leads to computation that grows exponentially with the number of unresolved positions.
Traditional alignment methods for discrete diffusion models assume that inference is unidirectional—once a token is sampled and determined it is not reconsidered—lacking a mechanism for the model to revisit already-determined choices.
Discrete diffusion models specifically handle discrete data (such as amino acid sequences, DNA sequences, or text); the forward process typically employs a masking mechanism that replaces certain positions in the sequence with [MASK] tokens, which the model progressively predicts and restores during the reverse process.
Protein generation models such as ESM3 and MDLM belong to the discrete diffusion model architecture.
BERT's MLM performs one-shot prediction and is primarily used for representation learning, whereas discrete diffusion models treat the masking ratio as a continuous variable from 0% to 100% to construct a complete Markov chain, achieving progressive generation through multi-step iteration.
All Facts (9)
When guiding discrete diffusion models with sequence-level goals, enumerating all possible completion combinations leads to computation that grows exponentially with the number of unresolved positions.
50%UnverifiedCOFFEE is a plug-and-play framework that adds sequence-level goal guidance to discrete diffusion models without retraining the pretrained model.
50%UnverifiedDiscrete diffusion models construct complete sequences by iteratively and in parallel resolving multiple tokens across multiple steps, unlike the left-to-right word-by-word generation of autoregressive models.
50%UnverifiedMDLM and SEDD are representative works in discrete diffusion models.
50%UnverifiedProtein generation models such as ESM3 and MDLM belong to the discrete diffusion model architecture.
50%UnverifiedTraditional alignment methods for discrete diffusion models assume that inference is unidirectional—once a token is sampled and determined it is not reconsidered—lacking a mechanism for the model to revisit already-determined choices.
50%UnverifiedDiscrete diffusion models specifically handle discrete data (such as amino acid sequences, DNA sequences, or text); the forward process typically employs a masking mechanism that replaces certain positions in the sequence with [MASK] tokens, which the model progressively predicts and restores during the reverse process.
50%UnverifiedBERT's MLM performs one-shot prediction and is primarily used for representation learning, whereas discrete diffusion models treat the masking ratio as a continuous variable from 0% to 100% to construct a complete Markov chain, achieving progressive generation through multi-step iteration.
50%Unverified扩散模型是一类基于马尔可夫链的生成模型，核心思想是通过前向加噪和反向去噪来捕捉数据分布
50%