JetSpec++: Forward-Looking Parallel Drafting and Self-Correction for Accelerated LLM Inference

The Bottleneck of Speculative Decoding: The Trade-off Between Draft Quality and Latency

Speculative Decoding has become a mainstream approach for accelerating Large Language Model (LLM) inference. Its core logic is to use a lighter draft model to generate several tokens in parallel first, then have the target model verify them in a single pass. Whether this paradigm works depends on two metrics that pull against each other: the acceptance rate of draft tokens, and the cost of producing those drafts.

Improving the acceptance rate typically means the draft model must be stronger and more expensive; while suppressing cost easily sacrifices quality, causing many tokens to be rejected by the target model and shrinking the speedup gains. For a long time, these two have been more of a trade-off than a win-win in engineering practice. The latest work from hao-ai-lab, JetSpec++, proposes that through algorithm-system co-design, these two objectives can be optimized simultaneously rather than as an either-or choice.

Acceptance Rate is the core metric for measuring speculative decoding efficiency, defined as the proportion of tokens generated by the draft model that are verified and accepted by the target model. Its mathematical expectation determines how many effective tokens can be produced per verification step, typically measured by "Average Acceptance Length" — the number of tokens successfully adopted in a single target model forward pass. If the draft is completely correct, in theory an entire draft block can be accepted in a single verification; once a position is rejected, all draft tokens after that position are discarded, because autoregressive generation depends on the accuracy of preceding tokens. Therefore, the impact of acceptance rate on speedup is non-linear: it determines both effective output and the degree of waste in the verification phase. This is precisely why the article emphasizes the "trade-off between draft quality and latency."

Wall-clock Time refers to the real physical time elapsed from when a request is issued to when the complete response is returned, as distinct from FLOPs or theoretical latency that only count computation. In inference acceleration scenarios, wall-clock time is the metric users actually perceive, and it is the ultimate goal that all of JetSpec++'s optimizations deliver on.

JetSpec++'s Three Major Components

JetSpec++'s starting point is straightforward: keep a strong learned draft model, but have it consume almost no extra wall-clock cost. To this end, the authors designed three interlocking mechanisms.

Intra-block Self Refinement

Traditional parallel drafting produces a token block in a single forward pass, with each position in the block predicted independently of the others, lacking opportunities for error correction. JetSpec++ trains the draft model to iteratively refine its own parallel predictions within a single token block, letting the draft model improve intra-block quality in a manner akin to "reviewing a few more times before finalizing." This directly acts on the acceptance rate — more accurate predictions mean more tokens accepted by the target model.

Inter-block Foresight Drafting

Before a token block has been verified, the model begins generating the next block. The significance of this step is to free draft generation from the chain of "serially waiting for verification results," keeping the draft model continuously busy, thereby amortizing the draft cost along the time axis.

Pipelined Drafting and Verification

What truly brings draft cost close to zero is overlapping the two stages of draft generation and target model verification. Since verification itself takes time, the drafting stage can be "hidden" within this time window, making the marginal wall-clock cost of drafting nearly negligible. The combined effect of the three components is to let the cost of a strong draft model be masked by system scheduling, while preserving a high acceptance rate.

Lossless Speculative Decoding means that speculative decoding, while accelerating, guarantees that the final generated token sequence is completely identical in probability distribution to the result of independent autoregressive sampling from the target model. Traditional speculative decoding achieves this through "Rejection Sampling" or "corrected sampling": the verification phase does not simply accept or reject draft tokens, but decides whether to accept based on the ratio of draft to target model probabilities, and resamples from a corrected distribution upon rejection. This way, even if draft model quality is limited, the output distribution is not distorted. JetSpec++ claims to "fully preserve the target model's output distribution," meaning all of its speedups are built on a strictly lossless premise, introducing no shift in generation quality or diversity.

Measured Performance: Up to 7.45x Speedup

The comparative data presented in the paper is quite convincing. Compared with current state-of-the-art speculative decoding methods, JetSpec++ achieves up to 7.45x wall-clock speedup relative to autoregressive (AR) decoding, and a 1.54x improvement over DSpark.

On the hardware side, JetSpec++ reaches up to 1,361 tokens/second generation speed per single request on GB200. Across seven benchmarks, the average acceptance length reaches 8.94 — this number indicates that the vast majority of tokens in draft blocks are adopted by the target model, direct evidence at the acceptance rate level. More critically, JetSpec++ fully preserves the target model's output distribution while accelerating, without changing the model's generation behavior for the sake of speed.

DSpark is a representative baseline method in the speculative decoding field, belonging to one of the routes of joint optimization between Draft and the target model. In JetSpec++'s comparison, DSpark is used as a reference as the current state-of-the-art (SOTA) method; JetSpec++'s 1.54x wall-clock speedup over it shows that its gains come not only from the framework dividends of classic speculative decoding, but also from the additional headroom brought by algorithm-system co-design.

GB200 is NVIDIA's next-generation GPU platform for large-scale AI inference and training, featuring higher memory bandwidth and denser compute units. JetSpec++ achieving 1,361 tokens/second per request on GB200 shows that its optimizations can still be fully unleashed on high-performance hardware.

Why This Matters

The engineering value of speculative decoding lies in "lossless acceleration," but its gains are highly dependent on the cost-effectiveness of the draft model. JetSpec++'s approach offers a reusable paradigm: do not sacrifice draft quality; instead, hide the draft cost through scheduling. Intra-block self refinement addresses quality, inter-block foresight and pipelining address cost, and the three correspond respectively to optimization objectives at the algorithm and system levels.

From a deployment perspective, the official team has open-sourced the code and model on GitHub (hao-ai-lab/JetSpec), which lowers the barrier to reproduction and secondary development. For teams needing to improve inference throughput under limited GPU resources, this kind of "algorithm + system" co-design approach may be more cost-effective than simply swapping in a larger draft model.

One note: the speedup ratios and acceptance lengths in the paper come from specific benchmarks and hardware configurations, and real-world deployment gains will vary with task type, batch size, and target model scale. The appearance of the open-source repository also gives the community an opportunity to validate the universality of these numbers under more real-world workloads.

One-line Summary

Through algorithm-system co-design, JetSpec++ lets a strong draft model consume almost no extra wall-clock cost, achieving up to 7.45x lossless speedup.

Paragraph Summary

Speculative decoding has long faced a trade-off between draft quality and draft cost: a stronger draft model yields a higher acceptance rate but also greater generation cost. JetSpec++ proposes algorithm-system co-design, preserving a strong learned draft model while hiding its cost within the target model's verification time window. The three components are intra-block self refinement (improving intra-block prediction accuracy), inter-block foresight drafting (keeping the draft model continuously busy), and pipelined drafting and verification (overlapping the two stages). Measured results show up to 7.45x speedup over autoregressive decoding, 1.54x over DSpark, up to 1,361 tokens/second per request on GB200, an average acceptance length of 8.94 across seven benchmarks, and full preservation of the target model's output distribution. Code and model are open-sourced on GitHub.