Speculative Decoding
An LLM inference acceleration technique formally proposed by Google DeepMind in 2023. A smaller draft model rapidly generates candidate tokens, which a larger target model then verifies in parallel. This approach can significantly reduce inference latency and computational cost.
Core Facts
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
Speculative decoding typically delivers a 2–3× speedup in practical applications.
Timeline (last 90 days)
Speculative decoding technology makes the speed approximately 1.6 to 1.8 times faster, and in actual tests the large model accepted the small assistant's guesses approximately 79% of the time
Strata uses the model's built-in multi-token prediction layer for speculative decoding, drafting up to 3 tokens at a time and producing an average of 2.4 to 3.2 tokens per pass, with the output results unchanged.
Techniques such as speculative decoding and Mixture of Experts architectures represent vendors' engineering efforts to compress inference costs while maintaining equivalent capability.
Compared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
In code generation scenarios, predictable patterns such as indentation, brackets, and repeated keywords are conducive to improving the acceptance rate of speculative decoding.
When setting up speculative decoding, too large a draft length can eat into gains due to more verification and rejected guesses, so one should start from auto/recommended settings.
Speculative decoding generates candidate tokens through a lightweight draft model, with the main model verifying them in parallel in a single forward pass, accepting tokens that conform to the distribution and regenerating from the first rejected position.
This test did not use any speculative decoding methods or MTP.
Under a deeply partitioned PP4 topology, speculative decoding cannot provide effective speedup because the communication cost of cross-GPU verification offsets the gains from batch verification.
Model quantization, speculative decoding, and continuous batching are software techniques used to reduce inference latency.
24 more timeline events
All Facts (20)
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
90%VerifiedSpeculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedSpeculative decoding typically delivers a 2–3× speedup in practical applications.
80%VerifiedWhen speed improvements and price reductions occur simultaneously, they typically point to underlying inference architecture optimization, with possible paths including model distillation, quantization, speculative decoding, and streaming processing architectures.
75%VerifiedSpeculative decoding is a lossless acceleration technique that does not alter the model's output distribution; it uses a draft model with far fewer parameters than the target model to generate candidate tokens, which the target model then verifies in parallel.
75%VerifiedSpeculative decoding was formally proposed by Google DeepMind in 2023; overall throughput can be improved by 2–3 times, and the output quality is completely equivalent to that of the original large model.
75%VerifiedCompared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
65%VerifiedSpeculative decoding submits the candidate sequence generated by the draft model to the target model as a whole for one-shot batch verification, requiring only a single forward pass to determine whether each candidate token is correct.
65%VerifiedSpeculative decoding was formally proposed by Google in 2023, using a draft model with 1/10 to 1/50 the parameters of the target model to generate multiple candidate tokens at once, which are then verified in parallel by the large model.
65%VerifiedSpeculative decoding uses a small local model to rapidly generate draft tokens, which are then verified and corrected by a large cloud-based model.
65%VerifiedThe combination of Multi-Token Prediction and speculative decoding can achieve a 2–4x inference speed improvement without compromising generation quality.
65%Verified推测解码（Speculative Decoding）是目前最主流的大模型推理加速路径之一，使用小型草稿模型快速生成候选token再由主模型并行验证
65%Unverified推测解码、模型蒸馏、KV Cache优化等推理优化技术的持续进步正在使单位推理成本快速下降
85%Unverified推测解码技术已被Google、Meta等公司广泛应用于生产环境
80%UnverifiedSpeculative decoding generates candidate tokens through a lightweight draft model, with the main model verifying them in parallel in a single forward pass, accepting tokens that conform to the distribution and regenerating from the first rejected position.
70%UnverifiedSpeculative decoding is a lossless LLM inference acceleration technique formally proposed by Leviathan et al. in 2023.
70%UnverifiedThe verification process in traditional speculative decoding accepts draft-proposed tokens with probability min(1, p(x)/q(x)), and upon rejection resamples from the corrected distribution norm(max(0, p(x)-q(x))), which can be proven to produce a token distribution identical to directly sampling from the target model (lossless).
70%UnverifiedLing 3.0 Tiny achieves an inference speed of approximately 86–90 tokens per second on a MacBook with M4 Pro, with a peak memory usage of approximately 8.34 GiB under an 8K context.
60%UnverifiedThe main optimization approaches in the industry for reducing LLM inference costs include model quantization, speculative decoding, and semantic caching mechanisms.
60%UnverifiedMTP technology can increase inference speed by approximately 2×.
60%