Why Does Speculative Decoding Speed Up LLM Inference?

Speculative Decoding has become a hot technique for accelerating large model inference in recent years, and a discussion thread on Hacker News titled Why Is Speculative Decoding Fast? has once again drawn attention. Although the thread itself did not see much interaction, the question gets at the core of the technology: how does it manage to speed up inference severalfold without sacrificing output quality?

The Basic Principle of Speculative Decoding

The core idea of speculative decoding is "guess first, verify later." It introduces a smaller draft model that quickly generates multiple candidate tokens, and then has the target large model verify in parallel whether these tokens are acceptable. Because the large model's forward computation can process multiple positions in parallel, the verification cost is far lower than generating them one by one.

The specific process consists of three steps: the draft model autoregressively generates K tokens; the target model performs one parallel forward pass over these K+1 positions; and based on the acceptance probability, it decides which tokens to keep and resamples at the first rejected position. The entire process guarantees that the output distribution is exactly identical to that of the target model, making it "lossless acceleration."

The serial bottleneck of autoregressive decoding: Traditional large model inference uses an autoregressive approach, where generating each token requires a complete forward pass with all previous tokens as input. This process is limited by memory bandwidth—each step must read all model weights from VRAM, while the actual computation is relatively small, leaving a large amount of GPU compute idle. The longer the sequence, the more serial steps there are, and the more latency accumulates. Speculative decoding targets exactly this bottleneck: since single-step compute utilization is low, it uses one forward pass to process multiple candidate positions in parallel, putting otherwise wasted compute to use, thereby significantly reducing serial steps while increasing total computation only modestly.

The mathematical guarantee of acceptance-rejection sampling: The losslessness of speculative decoding relies on a modified rejection sampling mechanism. For each token proposed by the draft model, the target model computes its probability and decides whether to accept or reject it accordingly; once rejected, it resamples from the difference (after normalization) between the target model and draft model distributions. This design makes the joint distribution of the finally accepted tokens strictly equal to the target model's autoregressive distribution, so the output is statistically indistinguishable from decoding with the target model alone. This is also the theoretical basis for the claim of "lossless acceleration," rather than an empirical approximation.

The Root of the Speedup: Parallel Verification and Acceptance Rate

What makes speculative decoding fast is the key point of converting "serial generation" into "parallel verification." Traditional autoregressive decoding requires a complete forward pass for each token generated, whereas speculative decoding can ideally verify multiple tokens with a single target model forward pass, thereby reducing the number of target model calls.

The speedup ratio mainly depends on two factors: the generation speed of the draft model and the acceptance rate of candidate tokens. The higher the acceptance rate, the more tokens can be confirmed per verification on average, and the more obvious the speedup. If the draft model's distribution is close to the target model's, the acceptance rate is high; conversely, a draft model that is too weak leads to frequent rejections and diminished speedup.

Why It Is Not Simply "Replacing the Large Model with a Small One"

A common misconception is: since small models are fast, why not just use a small model directly? The ingenuity of speculative decoding is that it uses the small model to "propose," but the final output is "vetted" by the large model. Accepted tokens strictly follow the target model's distribution, and when rejected, it falls back to the target model for resampling. Therefore, the output quality is exactly identical to decoding with the large model alone, and the small model is only responsible for providing efficient candidates.

This also explains why speculative decoding is mathematically lossless: the acceptance-rejection sampling mechanism ensures the correctness of the final distribution. The answer to "why is it fast" mentioned in the Hacker News discussion is essentially that "parallel verification amortizes the target model's computation cost."

Distilled models and small models from the same family: In practice, the choice of draft model directly affects the acceptance rate. Small models from the same family (such as a version with 1/10 the parameters of the large model) are a common choice because their training data and architecture are similar and their output distribution is relatively close to the target model's. Distilled models further improve alignment by having the draft model fit the target model's output distribution, thereby raising the acceptance rate.

The differing approaches of Medusa and EAGLE: Medusa adds multiple prediction heads on top of the target model to predict tokens at multiple subsequent positions in parallel, without needing an independent draft model, thus saving extra forward overhead; EAGLE uses the target model's feature layers for autoregressive draft generation, making predictions in feature space rather than token space, which usually achieves a higher acceptance rate. The common goal of such improvements is to make candidate tokens closer to the target model's distribution without significantly increasing computation.

Key Trade-offs in Practice

When deploying speculative decoding in practice, it is necessary to balance the size of the draft model and the acceptance rate. If the draft model is too small, the acceptance rate is low and verification is wasted; if the draft model is too large, the generation cost is high and offsets the parallel gains. Common practices include using small models from the same family, distilled models, or improved schemes such as Medusa and EAGLE to raise the acceptance rate through additional prediction heads or feature reuse.

Another dimension is batch size. Speculative decoding yields the greatest benefit in low-batch, memory-bandwidth-limited scenarios, because the target model's compute utilization is already low at that point, and parallel verification can fill the idle compute. In high-batch scenarios, the target model is already close to compute saturation, and the room for speedup narrows.

Implications for Developers

Speculative decoding is not a universal accelerator; its benefits depend heavily on the task type and model pairing. For tasks with strong output determinism (such as code completion and structured generation), the acceptance rate is usually higher; for open-ended creation, the acceptance rate may decline. When selecting a solution, developers should pay attention to acceptance-rate metrics and conduct actual tests in light of batch size and latency requirements.

Overall, the "speed" of speculative decoding comes from parallel verification breaking through the serial bottleneck, while the "stability" comes from the acceptance-rejection mechanism's guarantee of the output distribution. Understanding these two points makes it clear why it has become an important direction for large model inference optimization.

One-line Summary

Speculative decoding uses a small model to propose and a large model to verify in parallel, converting serial generation into parallel processing and achieving lossless acceleration.

Paragraph Summary

Speculative decoding introduces a draft model to quickly generate multiple candidate tokens, which are then verified in parallel in a single pass by the target large model, converting traditional token-by-token serial generation into parallel verification and thereby reducing the number of forward calls to the target model. Its speedup ratio depends on the draft model's speed and the candidate acceptance rate; the higher the acceptance rate, the more obvious the speedup. Because it uses an acceptance-rejection sampling mechanism, the final output distribution is exactly identical to decoding with the target model alone, making it lossless acceleration. It is not about replacing the large model with a small one, but about the small model proposing and the large model vetting. In practice, it is necessary to balance the draft model size and acceptance rate, and the greatest benefit is achieved in low-batch, memory-bandwidth-limited scenarios.