Continuous Diffusion Language Models: Can They Replace the Autoregressive Paradigm for Text Generation?

Language Models Are Exploring New Directions

The autoregressive Transformer architecture has nearly monopolized the technical roadmap for large language models — from the GPT series to various open-source models, all follow the "predict the next token one at a time" generation approach. The mathematical essence of autoregressive generation is decomposing the joint probability P(x₁,x₂,...,xₙ) into a product of conditional probabilities: P(x₁)·P(x₂|x₁)·P(x₃|x₁,x₂).... While this causal decomposition is mathematically rigorous and training-efficient (all position losses can be computed in parallel via teacher forcing), inference must be strictly sequential — each generated token requires a full forward pass, which is a key structural reason why inference costs for large models remain stubbornly high.

However, the tremendous success of Diffusion Models in continuous data domains like images and audio has given rise to an even more imaginative question: Can we use the diffusion model approach to generate text?

"Continuous Diffusion Language Models" (CDLM) represent exactly this line of exploration. Although attention to this area is still in its early stages, it touches on one of the most cutting-edge and controversial topics in generative AI: whether the diffusion paradigm can challenge the dominance of autoregression.

Continuous Diffusion Language Models Discussion Page

What Are Continuous Diffusion Language Models?

The Leap from Image Diffusion to Text Diffusion

The core idea behind diffusion models is "add noise, then denoise": during training, noise is progressively added to data until it becomes pure noise; during generation, the process starts from noise and gradually restores clear data. Mathematically, the forward process is a Markov chain that injects Gaussian noise into the data over T steps until the data distribution approaches a standard normal distribution; the reverse process trains a neural network to predict the noise component at each step (i.e., ε-prediction) or directly predict the denoised data. The key mathematical tools underpinning this mechanism include the variational lower bound (ELBO), score matching, and stochastic differential equations (SDEs). The introduction of DDPM (Denoising Diffusion Probabilistic Models) in 2020 enabled diffusion models to achieve GAN-competitive quality in image generation for the first time, and subsequent models like Stable Diffusion and DALL·E pushed this paradigm into large-scale commercial applications.

However, migrating diffusion models to the text domain faces a fundamental obstacle: images consist of continuous pixel values, while text consists of discrete tokens. Noise can naturally be superimposed on continuous pixels, but it's difficult to apply directly to discrete symbols — you can't add 0.3 standard deviations of Gaussian noise to the word "cat" to get a "slightly blurry cat."

The Clever Design of Continuous Embedding Spaces

The key to CDLM lies in the word "continuous." Rather than performing diffusion directly on discrete tokens, it maps text into a continuous embedding space, executes the diffusion process on this continuous representation, and then decodes back to discrete text.

An embedding space is a representational space that maps discrete symbols to high-dimensional continuous vectors. In NLP, each token is converted via an embedding matrix into a d-dimensional vector (typically d is 768, 1024, or higher), and these vectors have continuously differentiable semantic properties — semantically similar words are closer together in the embedding space. Score matching is one of the core training objectives of diffusion models, learning the gradient of the data distribution (i.e., the "score function" ∇log p(x)) rather than directly learning the probability density itself, cleverly avoiding the computational challenge of the normalization constant. CDLM leverages the continuity of the embedding space, allowing these mature mathematical tools originally designed for continuous data to be seamlessly reused.

Unlike autoregressive models that generate word by word, diffusion language models can iteratively refine the entire sequence in parallel — the initial output may be blurry or random, but after multiple rounds of denoising, it gradually converges into semantically coherent text.

Potential Advantages of Diffusion Language Models Over Autoregressive Models

Parallel Generation and Global Planning Capability

A major limitation of autoregressive models is sequential generation: earlier tokens must be produced before later ones can be generated, which limits inference speed and makes it difficult for the model to perform "global planning." Diffusion models naturally support parallel processing of the entire sequence, theoretically offering the following improvements:

Faster generation: With an appropriate number of sampling steps, reducing the per-word waiting time for long sequences.
Enhanced generation controllability: Since the entire sequence is iteratively refined, the model can more easily backtrack and correct during generation, enabling flexible tasks like infilling and constrained generation.
Better bidirectional context understanding: Unlike autoregressive models that can only see left-side context, diffusion models can leverage global information at every step.

Error Correction Through Iterative Denoising

The "multi-step denoising" nature of diffusion generation means the model has the opportunity to progressively correct errors. In contrast, once an autoregressive model generates an incorrect token, subsequent generation tends to be "led astray" by that error — this is academically known as the "exposure bias" problem, where the model sees real prefixes during training but may face its own erroneous prefixes during inference, causing errors to accumulate step by step. The iterative refinement property of diffusion models naturally mitigates this issue: each denoising step has the opportunity to globally review and correct imperfections from the previous step, giving diffusion language models a potential advantage in robustness.

Real-World Challenges Facing Continuous Diffusion Language Models

The Trade-off Between Generation Quality and Sampling Efficiency

Despite the appealing theoretical advantages, diffusion language models currently face severe challenges. First is the generation quality problem: at comparable parameter scales, the perplexity and downstream task performance of diffusion language models typically still lag behind mature autoregressive models. Perplexity is a classic metric for evaluating language model quality, defined as the reciprocal of the geometric mean of the model's predicted probabilities on the test set, i.e., PPL = exp(-1/N · Σlog P(xᵢ)). Intuitively, lower perplexity means the model is less "surprised" by real text — a model with perplexity of 10 is equivalent to facing an average of 10 equally probable options at each position. State-of-the-art autoregressive large language models already achieve very low perplexity on standard test sets, making the performance gap that diffusion language models need to bridge quite significant.

Second is the sampling efficiency bottleneck: although all positions can be processed in parallel within a single step, ensuring generation quality often requires dozens or even hundreds of denoising iterations, each requiring a full model forward pass. This means the actual speed advantage is not always obvious, especially when autoregressive models are equipped with acceleration techniques like KV Cache and speculative decoding, which may further narrow the speed gap.

The Decoding Challenge: From Continuous Representations to Discrete Tokens

While performing diffusion in embedding space is elegant, "how to precisely decode continuous representations back to discrete tokens" remains an error-prone step. Small deviations in embedding space can lead to completely wrong word decodings — for example, a denoised vector might land in the ambiguous zone between the embeddings for "cat" and "dog," and nearest-neighbor decoding may yield unstable results. This "quantization error" is further amplified as sequence length increases, and it's a focal point of community discussion.

Some researchers believe that discrete diffusion (defining the diffusion process directly on tokens) may be a more natural fit for text. Typical methods include D3PM (Discrete Denoising Diffusion Probabilistic Models) and MDLM (Masked Diffusion Language Models). D3PM implements "noising" by defining transition matrices between tokens — replacing a token with another token or a uniform noise distribution with certain probability; MDLM borrows from BERT's masking strategy, defining the forward process as progressively replacing tokens with [MASK], while the reverse process predicts the masked content. The advantage of discrete diffusion is that it completely avoids error accumulation from continuous-to-discrete conversion, but the challenge is that discrete spaces lack continuous gradient information, making sampling optimization more difficult. Recent research shows that carefully designed discrete diffusion models can already approach autoregressive model performance on certain tasks.

The Significance and Future Directions of Diffusion Language Models

The Exploratory Value of Breaking Paradigm Homogeneity

Regardless of whether CDLM ultimately surpasses autoregressive models in performance, the exploration itself is quite important. The current large language model field is highly homogenized, with nearly all mainstream models built on the same architecture and training paradigm. Diffusion language models represent a fundamentally different technical path, providing academia and industry with a valuable "second option" that helps prevent the entire field from getting stuck in the local optimum of a single approach.

Hybrid Paradigms: Autoregressive Meets Diffusion

It's worth noting that diffusion and autoregression are not mutually exclusive. Recent work has attempted to combine the two — for example, SUNDAE (Step-unrolled Denoising Autoencoders) introduces autoregressive-style unrolled training within the diffusion framework; Google's DART (Denoising Autoregressive Transformer) has the model generate a rough draft autoregressively, then refine it through a diffusion process. Another direction worth watching is "semi-autoregressive" generation — the model generates a chunk of tokens in parallel at each step, while chunks are still generated in autoregressive order, essentially seeking the optimal balance between parallelism and sequential dependency.

This hybrid paradigm may be the most promising direction for the future: combining the high-quality output of autoregressive models with the parallelism and controllability of diffusion models. These fusion attempts also hint at a possibility: the most powerful text generation systems of the future may not belong purely to any single paradigm, but rather flexibly combine the strengths of different generation mechanisms based on task characteristics.

Conclusion

Continuous diffusion language models are still in the early stages of research — they are neither an immediate replacement for autoregressive models nor anywhere close to maturity. But the question they raise is profound enough: Do we really need to generate text word by word? In an era of rapidly evolving generative AI, this kind of exploration that challenges the mainstream paradigm is precisely the force that pushes the boundaries of technology. For developers and researchers following the AI frontier, diffusion language models are a research direction well worth tracking.