JetSpec++
A speculative decoding acceleration method based on algorithm-system co-design. Through three components—intra-block self-correction, inter-block lookahead drafting, and pipelined drafting and verification—it substantially improves LLM inference speed while guaranteeing lossless output distribution.
Timeline (last 90 days)
Through algorithm-system co-design, JetSpec++ can simultaneously optimize draft token acceptance rate and draft cost, rather than choosing one over the other.
JetSpec++ comprises three major components: Intra-block Self Refinement, Inter-block Foresight Drafting, and Pipelined Drafting and Verification.
Intra-block Self Refinement trains the draft model to iteratively correct its own parallel predictions within a single token block, improving intra-block quality and thereby affecting the acceptance rate.
Inter-block Foresight Drafting starts generating the next block before a token block has been verified, keeping the draft model continuously busy to amortize draft cost.
Pipelined Drafting and Verification overlaps the two stages of draft generation and target model verification, making the marginal wall-clock cost of drafting nearly negligible.
JetSpec++ achieves up to a 7.45x speedup in wall-clock time relative to autoregressive (AR) decoding.
JetSpec++ achieves a 1.54x wall-clock time improvement relative to DSpark.
JetSpec++ reaches a generation speed of up to 1,361 tokens/second for a single request on NVIDIA GB200.
JetSpec++ achieves an average acceptance length of 8.94 across seven benchmarks.
While accelerating, JetSpec++ fully preserves the output distribution of the target model, achieving lossless speculative decoding.
2 more timeline events
All Facts (12)
The speedup ratios and acceptance lengths in the paper all come from specific benchmarks and hardware configurations, and the gains in actual deployment will vary with task type, batch size, and target model scale.
50%UnverifiedJetSpec++ comprises three major components: Intra-block Self Refinement, Inter-block Foresight Drafting, and Pipelined Drafting and Verification.
50%UnverifiedIntra-block Self Refinement trains the draft model to iteratively correct its own parallel predictions within a single token block, improving intra-block quality and thereby affecting the acceptance rate.
50%UnverifiedInter-block Foresight Drafting starts generating the next block before a token block has been verified, keeping the draft model continuously busy to amortize draft cost.
50%UnverifiedPipelined Drafting and Verification overlaps the two stages of draft generation and target model verification, making the marginal wall-clock cost of drafting nearly negligible.
50%UnverifiedJetSpec++ achieves up to a 7.45x speedup in wall-clock time relative to autoregressive (AR) decoding.
50%UnverifiedJetSpec++ achieves a 1.54x wall-clock time improvement relative to DSpark.
50%UnverifiedJetSpec++ reaches a generation speed of up to 1,361 tokens/second for a single request on NVIDIA GB200.
50%UnverifiedJetSpec++ achieves an average acceptance length of 8.94 across seven benchmarks.
50%UnverifiedWhile accelerating, JetSpec++ fully preserves the output distribution of the target model, achieving lossless speculative decoding.
50%UnverifiedThrough algorithm-system co-design, JetSpec++ can simultaneously optimize draft token acceptance rate and draft cost, rather than choosing one over the other.
50%UnverifiedThe official code and models of JetSpec++ have been open-sourced on GitHub under the repository name hao-ai-lab/JetSpec.
50%