Speculative Decoding
A large language model inference acceleration technique in which a smaller draft model predicts multiple tokens in advance, which are then verified in batch by the primary model, thereby increasing overall inference throughput.
Core Facts
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
The speculative decoding paradigm was proposed almost simultaneously by teams at Google Brain and DeepMind in 2023.
Timeline (last 90 days)
The essence of speculative decoding is to predict multiple tokens at low cost and then have the main model verify them in batch; the higher the hit rate, the more valid tokens produced per verification step.
Speculative decoding rapidly generates candidate tokens via a small draft model and then has the main model verify them in parallel, typically achieving a 2–4× speedup, with outputs that are mathematically equivalent to those produced without it and no loss in precision
Speculative Decoding uses a smaller draft model to quickly predict subsequent tokens, which are then batch-verified by the main model, significantly improving overall throughput; MTP is one implementation variant of this, and models such as Qwen support it natively.
Compared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
The time cost for the main model to verify N tokens in parallel is almost the same as verifying 1 token, and the overall throughput improves linearly with the draft model's hit rate.
Speculative decoding improves generation speed without changing output quality.
According to NVIDIA's official tests, the optimal decoding combination for Qwen3 27B on Jetson is NVFP4 quantization combined with speculative decoding.
Common technical approaches for achieving a 200x inference speed improvement include model distillation, dedicated inference chips/NPU acceleration, speculative decoding, and quantization compression.
The speedup ratio of speculative decoding is less ideal than for small and medium models when bandwidth has become a hard bottleneck or when the model exceeds 200B.
Cloud providers can narrow the latency gap for large models through speculative decoding, KV cache optimization, or dedicated inference hardware, but the degree of improvement is limited.
30 more timeline events
All Facts (20)
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
90%VerifiedSpeculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedThe speculative decoding paradigm was proposed almost simultaneously by teams at Google Brain and DeepMind in 2023.
90%VerifiedSpeculative decoding guarantees that the final output distribution is completely consistent with pure target-model generation through a rejection sampling protocol.
80%VerifiedMTP is a technique that allows the model to predict multiple subsequent tokens in a single forward pass, reducing the number of inference computation rounds.
80%VerifiedThe ability of a large model to actively infer design specifications from platform context technically falls under Contextual Reasoning, which internalizes industry common sense from massive amounts of data rather than relying on hard-coded rules.
75%VerifiedModel compression techniques such as INT8 quantization and channel pruning can increase inference speed by 2–4 times with a precision loss of less than 1% mAP
70%VerifiedSpeculative Decoding technology can deliver a 3–5x speed improvement compared to token-by-token serial generation.
70%VerifiedWhen the draft model prediction accuracy of Speculative Decoding reaches 60–80%, the overall inference speed can be improved by 2–3 times.
70%VerifiedCompared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
65%VerifiedThe Flash series reduces computational load by activating only a subset of parameters during each inference through technologies such as MoE architecture and speculative decoding.
65%VerifiedThe combination of Multi-Token Prediction and speculative decoding can achieve a 2–4x inference speed improvement without compromising generation quality.
65%Verified推测解码（Speculative Decoding）是目前最主流的大模型推理加速路径之一，使用小型草稿模型快速生成候选token再由主模型并行验证
65%UnverifiedMTP与DFlash在架构上有本质不同，但都属于投机解码技术谱系
85%Partially VerifiedA higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
80%UnverifiedMTP和Speculative Decoding两项技术理论上可以叠加使用
75%UnverifiedSpeculative decoding is a lossless LLM inference acceleration technique formally proposed by Leviathan et al. in 2023.
70%UnverifiedThe verification process in traditional speculative decoding accepts draft-proposed tokens with probability min(1, p(x)/q(x)), and upon rejection resamples from the corrected distribution norm(max(0, p(x)-q(x))), which can be proven to produce a token distribution identical to directly sampling from the target model (lossless).
70%UnverifiedThe actionable method for validating a coding model is to select a real repair task with tests, fix the repository version and tool permissions, have candidate models complete the same task, and compare first-pass rate, patch readability, per-run cost, and number of human takeovers.
70%UnverifiedStreaming inference involves inference optimization techniques such as speculative decoding and continuous batching.
70%