Speculative Decoding
An LLM inference acceleration technique formally proposed by the Google Brain team in 2023. It introduces a small draft model to rapidly generate candidate token sequences, which are then verified in parallel by the main model, theoretically improving generation speed by a factor of two to four.
Core Facts
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
NVIDIA's TensorRT-LLM deeply optimizes inference speed for its hardware architecture.
Speculative decoding typically delivers a 2–3× speedup in practical applications.
Timeline (last 90 days)
Janas supports speculative decoding: a small draft model guesses the next few tokens and the large model verifies them in one pass. On Qwen3-Next-80B this raises write speed from 24 to 37 tok/s, with the text content completely identical.
Inference-side engineering optimizations such as quantization, speculative decoding, and PagedAttention are becoming the core means by which vendors widen the cost gap.
约束解码通过语法掩码保证输出符合JSON等文法，但会破坏MTP、投机解码等并行验证加速路径，导致速度下降，需在可解析性与吞吐速度之间权衡。
Inference engineering techniques such as speculative decoding and KV cache optimization are no less important to RSI research than the algorithms themselves, because halving generation speed means halving the number of iteration rounds within the same budget.
Common technical approaches for reducing the TCO of video AI systems include quantization-based model compression, speculative decoding to accelerate text generation, and dynamic routing to models of different scales based on content complexity (cascaded inference).
As computing costs decline and inference acceleration technologies such as model quantization and speculative decoding become widespread, the generation speed for equivalent quality is expected to be significantly reduced within the next one to two years.
Speculative decoding generates candidate tokens through a lightweight draft model, with the main model verifying them in parallel in a single forward pass, accepting tokens that conform to the distribution and regenerating from the first rejected position.
Integrating watermarking with speculative decoding addresses the pain point of watermarks being lost during acceleration.
v0.29.1rc0 introduces a Dual-key Gumbel-Max watermarking mechanism for speculative decoding.
Speculative decoding and Flash Attention are continuously emerging software optimization techniques.
21 more timeline events
All Facts (20)
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedNVIDIA's TensorRT-LLM deeply optimizes inference speed for its hardware architecture.
85%VerifiedSpeculative decoding typically delivers a 2–3× speedup in practical applications.
80%VerifiedMTP is a technique that allows the model to predict multiple subsequent tokens in a single forward pass, reducing the number of inference computation rounds.
80%VerifiedSpeculative decoding submits the candidate sequence generated by the draft model to the target model as a whole for one-shot batch verification, requiring only a single forward pass to determine whether each candidate token is correct.
65%VerifiedSpeculative decoding's rejection-sampling-based verification algorithm ensures that the output distribution is exactly equivalent to—rather than merely an approximation of—running the main model alone.
65%VerifiedThe combination of Multi-Token Prediction and speculative decoding can achieve a 2–4x inference speed improvement without compromising generation quality.
65%Verified推测解码（Speculative Decoding）是目前最主流的大模型推理加速路径之一，使用小型草稿模型快速生成候选token再由主模型并行验证
65%Unverified推测解码、模型蒸馏、KV Cache优化等推理优化技术的持续进步正在使单位推理成本快速下降
85%Partially VerifiedA higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
80%UnverifiedSpeculative decoding generates candidate tokens through a lightweight draft model, with the main model verifying them in parallel in a single forward pass, accepting tokens that conform to the distribution and regenerating from the first rejected position.
70%UnverifiedA reasonable prediction loss combined with 0% downstream control accuracy is highly consistent with representation collapse, meaning the encoder has fooled the self-supervised objective without learning any useful dynamic information.
70%UnverifiedSpeculative decoding generates drafts with a small model and verifies them in batch with a large model; mathematically it guarantees output fully consistent with the large model through rejection sampling, making it lossless acceleration
70%UnverifiedJanas supports speculative decoding: a small draft model guesses the next few tokens and the large model verifies them in one pass. On Qwen3-Next-80B this raises write speed from 24 to 37 tok/s, with the text content completely identical.
50%UnverifiedInference-side engineering optimizations such as quantization, speculative decoding, and PagedAttention are becoming the core means by which vendors widen the cost gap.
50%Unverified约束解码通过语法掩码保证输出符合JSON等文法，但会破坏MTP、投机解码等并行验证加速路径，导致速度下降，需在可解析性与吞吐速度之间权衡。
50%UnverifiedInference engineering techniques such as speculative decoding and KV cache optimization are no less important to RSI research than the algorithms themselves, because halving generation speed means halving the number of iteration rounds within the same budget.
50%UnverifiedCommon technical approaches for reducing the TCO of video AI systems include quantization-based model compression, speculative decoding to accelerate text generation, and dynamic routing to models of different scales based on content complexity (cascaded inference).
50%UnverifiedIntegrating watermarking with speculative decoding addresses the pain point of watermarks being lost during acceleration.
50%UnverifiedSpeculative decoding and Flash Attention are continuously emerging software optimization techniques.
50%