Autoregressive Model

The VRAM occupied by KV Cache grows linearly with sequence length.

vLLM has previously primarily served autoregressive large language models.

Acceleration solutions such as consistency models and flow matching are narrowing the computational overhead gap of multi-step inference in diffusion models.

Audio is a time-series signal, and music generation models need to maintain coherence over long time spans; integrating them into a workflow engine primarily designed for images/video requires specialized adaptation in sampling scheduling, memory management, and output format

Autoregressive models generate token by token, and if they miscount characters or miss a constraint at some intermediate step, the error accumulates forward and causes an overall collapse.

The control signal of autoregressive models can only be applied at the beginning of the sequence, that is, prefix prompting or prefix fine-tuning.

Autoregressive generation is a strictly left-to-right unidirectional process, making it difficult to precisely patch an error at an intermediate step without affecting subsequent content.

Unlike autoregressive models that strictly generate from left to right, MDLMs theoretically allow parallel filling of arbitrary positions.

The inference time of classification models is almost independent of the number of categories, whereas the latency of autoregressive generative models is proportional to output length.

Discrete diffusion models construct complete sequences by iteratively and in parallel resolving multiple tokens across multiple steps, unlike the left-to-right word-by-word generation of autoregressive models.

Traditional autoregressive models typically require additional processing or approximation methods to obtain reliable probability distributions, giving diffusion models a structural advantage in this regard.

The generation speed of autoregressive models is limited by their sequential, one-token-at-a-time nature.

The Prefill stage of autoregressive model inference processes the entire prompt at once, which is a standard GEMM operation with high arithmetic intensity and is compute-bound.

An autoregressive model requires only one forward pass to generate each token.

Autoregressive models suffer from the exposure bias problem, whereby the model sees real prefixes during training but faces self-generated erroneous prefixes during inference, causing errors to accumulate progressively.

There are currently two main technical approaches in the video generation field: a diffusion-model-based approach (such as Sora and H3) and an autoregressive-model-based approach; the former has greater advantages in image quality and motion coherence, while the latter has more potential in multimodal understanding and flexibility.

The current mainstream text-to-image models are primarily divided into three major technical approaches: diffusion models, Generative Adversarial Networks (GANs), and autoregressive models, with diffusion models having become the current mainstream due to their stable generation quality and relatively easy training.

Splicing autoregressive text generation together with diffusion-based iterative denoising introduces a fundamental asymmetry in computational paradigms.

Autoregressive models formulate text generation as a chain of conditional probabilities; generating a text of 1,000 tokens requires 1,000 forward inference passes, and the computational load of each inference increases as the context window grows.

On general text generation tasks, the fluency and coherence of mainstream diffusion models have not yet comprehensively surpassed those of autoregressive models of comparable scale.