Acceptance Rate ≠ Speedup: The "Length Inflation" Trap in Lossy Speculative Decoding

Speculative Decoding has long been regarded as a powerful tool for accelerating large model inference: a small model rapidly drafts tokens, which the large model then verifies in parallel, ideally delivering significant speedup without degrading output quality. However, a study on training-free lossy speculative decoding reveals a neglected phenomenon—relaxing the verification rule increases the number of tokens accepted per round, but causes generated outputs to become longer, thereby offsetting the seemingly substantial speedup gains.

The Misread Efficiency Metric: Tokens Accepted per Round

Under lossless verification, the output distribution exactly matches the target model, so "tokens accepted per round" genuinely reflects the true speedup ratio. But once relaxed (lossy) verification is adopted—accepting some tokens that lossless verification would have rejected—the output distribution is altered, and this metric becomes distorted.

The research team systematically compared existing training-free relaxed verification rules against lossless verification across twelve target model–draft model pairings with identical prompts, measuring completion length, number of verification rounds, actual wall-clock time, and task accuracy. The results are sobering: under the most permissive setting of each rule, relaxed verification achieved an average of 1.34× tokens accepted per round, but completion length inflated to 1.54×.

The difference between lossless and relaxed (lossy) verification lies in the criterion by which draft tokens are accepted by the target model. Lossless verification (e.g., standard speculative sampling) employs exact accept-reject sampling: the acceptance probability is the ratio of target to draft model probabilities, and rejected positions are resampled according to the residual distribution, so the final output distribution strictly matches the target model. Relaxed verification loosens this criterion—for example, accepting a draft token directly whenever the target model's probability exceeds some threshold, or using top-k/top-p approximate matching—without strict importance correction. This allows more tokens to be swallowed per round, but the accepted tokens no longer follow the target distribution, and output quality and behavioral characteristics may deviate from the original model. Understanding this distinction is the prerequisite for judging "whether acceptance rate can represent speedup ratio": only when the distribution is unchanged does the number of accepted tokens bear a monotonic relationship to true speedup.

Where Does Length Inflation Come From

The core mechanism behind this phenomenon is that tokens additionally accepted under lossy verification often have very low probability in the target model. Once these low-probability tokens enter the reasoning trajectory, they induce the model to continue "thinking" rather than converge on an answer, significantly lengthening the reasoning process.

In other words, the increase in acceptance rate does not come from more efficient parallel verification, but from the model being led down a longer reasoning path. This effect is especially pronounced for reasoning tasks—what relaxed verification mainly extends is precisely the reasoning traces.

The study also found that rules capable of reducing the number of verification rounds are typically accompanied by a marked drop in accuracy; some failure cases can be traced to the model continuing to reason, exhausting the budget that should have been used to produce the final answer, and ultimately failing to converge to a conclusion.

A reasoning trace refers to the series of intermediate reasoning steps a chain-of-thought-capable model generates before giving its final answer—such as decomposing the problem, trial and error, self-checking, etc. Unlike ordinary dialogue output, the length of a reasoning trace is not fixed; the model dynamically weighs "continuing to think" against "converging to answer": when it deems its current line of thought uncertain, it tends to keep expanding; only when confidence is sufficient does it output a conclusion. The longer the reasoning trace, the more generation budget is consumed (e.g., the maximum token limit), and end-to-end latency rises accordingly. Precisely for this reason, any mechanism that alters the token sequence distribution—including the extra low-probability tokens accepted by lossy verification—may change the model's judgment of "whether to keep thinking," thereby systematically affecting trace length. This is the deeper reason why evaluating speculative decoding cannot look only at throughput but must also attend to output length.

Implications for Engineering Practice

This analysis offers clear evaluation recommendations: in lossy speculative decoding scenarios, reporting only the number of tokens accepted per round is insufficient to measure decoding speed. To truly evaluate efficiency, one should at minimum also report completion length and number of verification rounds, and make a comprehensive judgment combined with actual wall-clock time and task accuracy.

For teams currently deploying speculative decoding, this means that model selection cannot rely solely on "acceptance rate." A relaxed rule with a higher acceptance rate may lag on both end-to-end latency and output quality. Truly valuable speedup should be built on the foundation of a controllable output distribution and stable completion length.

Conclusion

The speedup promise of speculative decoding does not hold unconditionally. When verification is relaxed, acceptance rate degrades from an efficiency proxy into an easily misleading number. The length inflation phenomenon reminds us: measuring inference speedup must ultimately return to the real performance of end-to-end latency and task quality.

One-line Summary

Lossy speculative decoding raises acceptance rate but lengthens outputs, offsetting speedup gains.

Paragraph Summary

This article dissects a neglected trap in training-free lossy speculative decoding: relaxed verification increases tokens accepted per round but simultaneously lengthens outputs, canceling out seemingly substantial speedup. Comparing relaxed verification rules against lossless verification across twelve target-draft model pairings with identical prompts, the study finds that under the most permissive settings tokens accepted per round reach 1.34×, while completion length inflates to 1.54×. The mechanism: extra-accepted low-target-probability tokens induce the model to keep reasoning rather than converge, especially on reasoning tasks; rules that reduce verification rounds often come with accuracy drops, and some failures stem from reasoning exhausting the budget before a final answer. The article accordingly recommends that in lossy scenarios, tokens accepted per round cannot alone measure speed—completion length, verification rounds, actual wall-clock time, and task accuracy must all be reported.