ProVe
An inference acceleration method that reverses the role assignment in speculative decoding. A large model is fine-tuned to act as the proposer, predicting multiple future tokens in a single forward pass, which are then verified with an additional forward pass, without requiring an extra draft head.
Timeline (last 90 days)
ProVe makes the large model itself the proposer, predicting multiple future tokens in one forward pass, then completing verification with one additional forward pass.
ProVe achieves an average 2.5x throughput improvement on 32B models without training an additional draft head.
On the GSM8K dataset, with a block size of 16, the average matched prefix length between ProVe proposal blocks and AR decoding is 6.6 tokens.
When a 3B small verifier is introduced, ProVe reaches an average 2.7x throughput, but its average accuracy across six tasks is 2.6 percentage points lower than AR.
ProVe is evaluated on six benchmarks using a 32B proposer.
ProVe's core innovation is fine-tuning the large model itself into a proposer for block-level prediction, avoiding training an independent draft head and simplifying deployment.
The small verifier (3B) is a lossy acceleration mode whose scoring criterion is not fully consistent with the proposer, potentially causing output distribution shift and accuracy degradation.
All Facts (7)
ProVe makes the large model itself the proposer, predicting multiple future tokens in one forward pass, then completing verification with one additional forward pass.
50%UnverifiedProVe achieves an average 2.5x throughput improvement on 32B models without training an additional draft head.
50%UnverifiedOn the GSM8K dataset, with a block size of 16, the average matched prefix length between ProVe proposal blocks and AR decoding is 6.6 tokens.
50%UnverifiedWhen a 3B small verifier is introduced, ProVe reaches an average 2.7x throughput, but its average accuracy across six tasks is 2.6 percentage points lower than AR.
50%UnverifiedProVe is evaluated on six benchmarks using a 32B proposer.
50%UnverifiedProVe's core innovation is fine-tuning the large model itself into a proposer for block-level prediction, avoiding training an independent draft head and simplifying deployment.
50%UnverifiedThe small verifier (3B) is a lossy acceleration mode whose scoring criterion is not fully consistent with the proposer, potentially causing output distribution shift and accuracy degradation.
50%