Diffusion Language Model

Diffusion language models are still at an early stage and continue to lag behind mature autoregressive models in generation quality, long-text coherence, and training stability.

The differentiated advantages of diffusion models manifest in two dimensions — repair success rate and generation latency — rather than in one-shot generation accuracy.

The arXiv paper "Plan-and-Patch: Diffusion Language Models for Agentic Planning" proposes using diffusion language models to achieve agentic planning and local repair.

Diffusion language models generate structured, program-like plans in a single pass via parallel unmasking.

Based on a masked-filling mechanism, diffusion models can retain prefixes and suffixes and conditionally refill excised middle segments, enabling local repair.

CoDR (Confidence Drift Remasking) is a training-free, sampler-agnostic method for correcting early errors in diffusion language models.

In a single one-step denoising step, the model directly reads out the probability distribution for each answer slot rather than sampling token by token.

Diffusion models natively support parallel prediction across multiple positions, enabling the probability distributions of all answer slots to be obtained at once.

Traditional autoregressive models typically require additional processing or approximation methods to obtain reliable probability distributions, giving diffusion models a structural advantage in this regard.

Diffusion language models have historically underperformed compared to mature autoregressive models of similar scale on mainstream benchmarks.

Diffusion language models adopt a parallel denoising approach, which can simultaneously optimize outputs at multiple positions in a single forward pass and then iteratively converge to the final text through multiple rounds.

The Transformer in diffusion language models typically employs bidirectional attention and needs to receive timestep information (injected via sinusoidal encoding or AdaLN), so standard pre-trained language model weights cannot be directly reused.

The number of iterative denoising steps is a tunable hyperparameter; more steps generally yield higher generation quality but take longer, while fewer steps are faster, providing a quality-efficiency trade-off.

On general text generation tasks, the fluency and coherence of mainstream diffusion models have not yet comprehensively surpassed those of autoregressive models of comparable scale.

Amazon's AGI team released a diffusion language model, open-sourcing weights in 1.7B and 8B parameter sizes along with training and inference code, with the most difficult tokens subject to up to four rounds of iterative refinement.