ProVe Decoding: When the Large Model Becomes the Proposer — Speculative Decoding in Reverse

Speculative Decoding typically uses a small model to quickly draft tokens, which are then verified in parallel by a large model, thereby accelerating autoregressive (AR) generation. But a new study called ProVe flips this role assignment: the large model itself acts as the proposer, predicting multiple future tokens in a single forward pass, followed by one additional forward pass for verification. This seemingly simple reversal achieves an average throughput improvement of 2.5x on a 32B model without training an additional draft head.

Core Idea: Reversing the Roles of Speculative Decoding

Traditional speculative decoding relies on an independent, smaller draft model to serially generate candidate tokens, which are then verified in parallel by the large model. While effective, this approach requires training and maintaining an additional draft model, increasing engineering complexity. ProVe instead starts directly from a standard AR checkpoint and fine-tunes the large model into a "Proposer," enabling it to predict a block of future tokens in a single forward pass. A subsequent additional forward pass then verifies this proposed block.

The verifier can be another smaller AR model, or it can be the proposer itself. The latter is called ProVe-lossless, as it can exactly reproduce the proposer's greedy AR decoding results. This means substantial speedup can be achieved with complete losslessness.

Autoregressive (AR) generation is the standard inference method for current large language models: the model predicts only the next token in each forward pass, then appends that token to the input sequence before executing the next forward pass. This serial process requires a full forward computation for each token, making inference latency proportional to generation length and a primary bottleneck for throughput.

Draft head refers to a lightweight prediction module attached to the model, typically trained separately, used to quickly generate candidate tokens in speculative decoding. Traditional methods require training and maintaining an independent draft model or draft head, increasing engineering complexity and deployment costs. ProVe avoids this extra component by fine-tuning the large model itself to predict token blocks.

Performance: 2.5x Throughput and Controllable Trade-offs

The researchers evaluated ProVe on six benchmarks. With a 32B proposer, ProVe-lossless achieved an average 2.5x AR throughput without training a separate draft head. The proposed block itself already provides sufficient acceleration: with a block size of 16, the average matching prefix length between the proposed block and AR decoding on the GSM8K dataset is 6.6 tokens.

For higher speed, a small 3B verifier can be introduced, where ProVe achieves an average throughput of 2.7x, at the cost of a 2.6 percentage point drop in average accuracy across six tasks compared to AR. This offers a controllable trade-off: users can flexibly choose between speed and accuracy based on their application scenario.

Matching prefix length is a key metric for measuring the acceleration effect of speculative decoding: it refers to the number of consecutive tokens from the beginning of the proposed block that match the target model's (or AR decoding's) output. The longer the matching prefix, the more tokens can be accepted at once during verification, resulting in a higher speedup ratio. In ProVe, with a block size of 16, the average matching prefix on GSM8K is 6.6 tokens, meaning each proposal-verification cycle can accept approximately 6.6 tokens on average, compared to 1 token in traditional AR.

Throughput typically refers to the number of tokens generated by the model per unit time (tokens/s), a core metric for measuring inference efficiency. A 2.5x throughput improvement means that under the same hardware conditions, ProVe generates 2.5 times more tokens per second than standard AR decoding, directly reducing the inference cost per token.

Technical Details and Significance

The key innovation of ProVe lies in transforming the large model itself into the proposer rather than relying on an external draft model. This avoids the overhead of training an independent draft head and simplifies the deployment process. Fine-tuning the large model to possess block-level prediction capability is the core of the method. The verification stage maintains standard parallel verification logic, ensuring controllable output quality.

From a broader perspective, ProVe demonstrates the flexibility of the speculative decoding framework: the roles of proposer and verifier are not fixed; by reassigning roles, acceleration can be achieved without adding extra models. This provides a new path for teams looking to reduce inference costs without introducing a complex draft model system.

Of course, the speedup ratio of ProVe-lossless is limited by the degree of matching between the proposed block and AR decoding. An average match of 6.6 tokens on GSM8K means there is still room for improvement. How to further increase the hit rate of the proposed block will be a direction for future optimization.

ProVe-lossless — "lossless" here means that when the verifier and proposer are the same model, the final output distribution is exactly identical to the proposer's standard greedy AR decoding, introducing no accuracy loss. The principle is: during verification, each token in the proposed block is scored in parallel, and only prefixes that match the proposer's own AR predictions are accepted, thereby ensuring the output sequence is the same as token-by-token greedy decoding.

Small verifier (3B) is a lossy acceleration mode: the verifier is an independent small model whose scoring criteria are not entirely consistent with the proposer, so it may accept tokens that the proposer would not choose, causing a shift in output distribution that manifests as reduced accuracy (2.6 percentage points lower on average across six tasks). This design allows users to trade an acceptable level of accuracy loss for higher throughput.

One-line Summary

ProVe reverses the roles of speculative decoding, letting the large model propose and verify, achieving a 2.5x throughput improvement on a 32B model.

Paragraph Summary

ProVe introduces a new method that reverses the traditional role assignment in speculative decoding: instead of using an independent small draft model, it fine-tunes the large model itself into a proposer that predicts multiple future tokens in a single forward pass, followed by one additional forward pass for verification. The verifier can be the proposer itself (ProVe-lossless, with output exactly matching greedy AR decoding) or a small 3B model (lossy but faster). On a 32B model, ProVe-lossless achieves an average 2.5x throughput improvement without training an extra draft head; introducing a 3B verifier raises throughput to 2.7x, but the average accuracy across six tasks drops by 2.6 percentage points. The method simplifies deployment and provides a controllable trade-off between speed and accuracy, while its speedup ratio is limited by the matching degree between the proposed block and AR decoding (an average match of 6.6 tokens with block size 16 on GSM8K).