Speculative Forcing: A New Framework for Accelerating Autoregressive Video Generation Inference

The Efficiency Dilemma of Autoregressive Video Generation

Autoregressive video generation models are changing the way we create dynamic content. By generating token by token, these models enable streaming video synthesis, making real-time content creation possible. However, the efficiency problem has always been a sword of Damocles hanging overhead.

Current mainstream approaches use few-step autoregressive generators to accelerate inference, but even so, each token still needs to pass through the full final denoising step, regardless of how much refinement it actually requires. This "one-size-fits-all" computation pattern causes massive unnecessary compute waste.

Research from rss source, 《Speculative Forcing for Efficient Autoregressive Video Generation》, reveals a key observation: many tokens are already very close to their final full-step output at the initial step, and only some tokens need continuous refinement until the very end. This means there is enormous room for optimization in uniform computation strategies.

What is autoregressive video generation?

The core idea of autoregressive generation originates from language models: break a piece of content into a sequence, predict only the next unit each time, and use the already-generated units as conditions for continued prediction. In video generation, this "unit" is typically a visual token after spatiotemporal compression—a discrete or continuous representation obtained by quantizing video frame blocks with a VQ-VAE or similar encoder. Compared with diffusion models that denoise an entire video at once, the autoregressive approach naturally supports streaming synthesis: it can generate and play simultaneously while the user waits, and can maintain relatively constant VRAM usage for ultra-long videos. The cost is that inference must proceed sequentially, and unlike image diffusion, all tokens cannot be denoised in parallel, so the computation cost of each token directly accumulates into a latency bottleneck. This is precisely the root of the "efficiency dilemma."

The Core Mechanism of Speculative Forcing

Treating Intermediate Estimates as "Drafts"

The core idea of the Speculative Forcing framework proposed by the research team is to treat the intermediate clear estimates in the denoising process as "drafts," and predict which tokens can be safely accepted early.

Once a token is accepted, it is frozen as the final output and no longer participates in subsequent denoising computation. Meanwhile, its cached keys and values are still used as context for the remaining tokens. This design both saves computation and maintains generation quality.

Few-step generation and denoising steps

Diffusion or flow-matching generative models usually require dozens of iterative denoising steps to obtain clear results. Few-step methods compress this process to 2 to 4 steps through distillation, consistency models, or adversarial training, allowing a single generation to complete within acceptable latency. However, few-step does not mean "every step is necessary": during actual sampling, different tokens converge at very different rates; some are already settled at the first step, while others only stabilize at the last step. Speculative Forcing adds another layer of adaptive pruning on top of the few-step generator—turning "the number of steps that must be completed" from a fixed value into a value dynamically determined per token, which in theory can reduce the average number of steps far below the configured upper limit. Understanding this is necessary to see why it can maintain four-step quality while reducing the average to about 1.7 steps.

Two Verifier Design Paths

The research explored two different verifier implementation approaches:

Training-free verifier: judges based on the similarity between the token and clear content as well as stability across steps. If a token remains stable across multiple steps and is sufficiently close to the clear content, it can be accepted early.

Learned verifier: reads the intermediate features of the frozen generator and estimates the remaining gap between each token and its full-step output. This method can more precisely predict which tokens have already "matured."

What is a verifier?

In a speculative generation framework, the verifier is responsible for answering a judgment question: "Can this token be finalized now?" It does not participate in generation itself, but only evaluates the maturity of the current intermediate result. The training-free verifier relies on hand-designed heuristic metrics, such as the magnitude of change in token representation across consecutive steps and similarity to the current clear estimate; its advantages are zero additional parameters and plug-and-play use. The learned verifier trains a small network that reads the intermediate-layer features of the frozen generator and directly regresses the error between that token and the final full-step output. The two form a trade-off between accuracy and cost: the former is faster but makes coarser judgments, while the latter is more accurate but requires additional training data and inference overhead. The quality of the verifier's judgment directly determines the acceleration ratio and image-quality loss, making it the key component of the entire framework.

Measured Performance Across Multiple Scenarios

Short Video and Long Video Generation

On the LongLive benchmark, Speculative Forcing delivered impressive results. It can match the quality of a four-step generator, but averages only about 1.7 denoising steps per token, improving runtime speed by more than 1.5x and reaching generation frame rates of 41.7 and 37.8 fps.

This means that, while maintaining visual quality, inference efficiency achieves a substantial improvement. For application scenarios that require real-time video generation, such an acceleration ratio has important practical value.

World Model Generation

Testing on Lingbot-World 2.0 is more challenging—camera motion continuously changes the visible content, imposing higher dynamic adaptation requirements on the generative model. In this scenario, Speculative Forcing still maintains quality, averaging about 2.5 denoising steps per token, improving runtime speed by 1.35x and reaching 24.5 fps.

You may not have noticed that in world generation scenarios the average number of steps required per token is higher (2.5 vs. 1.7), reflecting the higher refinement demands of dynamic content. But even so, the framework still found significant room for optimization.

Implications for the Field of Video Generation

The value of Speculative Forcing lies not only in its specific acceleration numbers, but also in the important design principle it reveals: not all tokens require the same amount of computational investment.

This idea has an affinity with speculative decoding in natural language processing, but it has been specially designed for the spatiotemporal characteristics of video generation. By using token "maturity" as the basis for computation allocation, the model can concentrate compute on the parts that truly need refinement.

From an engineering perspective, the strategy of retaining cached keys and values is also worth noting. Although accepted tokens no longer participate in denoising, their keys and values still serve as context—meaning that early acceptance decisions do not break the generation conditions for subsequent tokens.

As video generation models move toward longer durations and higher resolutions, the importance of inference efficiency will only continue to grow. The adaptive computation allocation approach provided by Speculative Forcing may become an important component of future video generation systems.

For researchers and engineers focused on the efficiency of AI video generation, the core insight of this framework—let each token compute on demand—is worth validating and extending across more architectures and tasks.

The analogy to speculative decoding

Speculative decoding is a mature acceleration technique in natural language processing: first, a small and fast draft model guesses several tokens in parallel, then a large model verifies them all at once, accepting the correct parts and rejecting the incorrect parts. It turns "token-by-token serial" processing into "batch guessing plus verification," improving throughput without changing the output distribution. Speculative Forcing borrows this conceptual skeleton of "draft first, verify later," but the object it faces is different—the "correctness" of a video token is not compared against a target distribution, but against the degree of convergence of its own full-step output; and accepted tokens still remain in the context in key-value form. This adaptation makes it suitable for video generation tasks with higher spatiotemporal coherence requirements, and also explains why world model scenarios require more conservative acceptance strategies.

One-line Summary

Speculative Forcing finalizes already-mature tokens early in video generation, allocating compute on demand to accelerate inference.

Paragraph Summary

Speculative Forcing is an acceleration framework for autoregressive video generation, improving on the wastefulness of existing few-step generators in which "every token must complete all denoising steps." Its core idea is to treat intermediate clear estimates during denoising as drafts, use a verifier to determine which tokens are mature enough, and once accepted, freeze them as final outputs, skip remaining computation, and retain their keys and values as context for subsequent tokens. The research presents both training-free and learned verifier approaches. In tests, it requires only about 1.7 denoising steps per token on the LongLive benchmark on average, achieves more than 1.5x speedup and 41.7 fps, and matches the quality of a four-step generator; on the world model Lingbot-World 2.0, it achieves 1.35x speedup and 24.5 fps with about 2.5 steps. Its core principle is to adaptively allocate compute according to token maturity.