AMTP
A lightweight online controller that dynamically selects the speculative horizon for multi-token prediction. Without retraining the model, it learns from verification feedback the optimal draft length for the current context.
Timeline (last 90 days)
AMTP (Adaptive Multi-Token Prediction) is a lightweight controller that requires no model retraining and does not alter the exact verifier.
AMTP uses the acceptance survival probability to characterize the expected number of tokens committed under each horizon, and selects the horizon with the highest expected tokens per second.
AMTP formulates horizon selection as an online control problem, learning the optimal horizon online through verification feedback.
The paper theoretically proves that AMTP's prediction error translates into throughput regret relative to the best horizon for each context, and that this regret is sublinear when online gradient descent is used.
Across five draft-target pairs built from Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, AMTP achieves up to 4.1x speedup over standard decoding.
In evaluations on AIME 2025 and GSM8K, AMTP matches or exceeds the fixed-window method in 8 of 10 settings, with throughput gains of up to 10%.
AMTP keeps the verifier unchanged and is compatible with existing exact verification pipelines, lowering the deployment barrier.
All Facts (7)
AMTP (Adaptive Multi-Token Prediction) is a lightweight controller that requires no model retraining and does not alter the exact verifier.
50%UnverifiedAMTP uses the acceptance survival probability to characterize the expected number of tokens committed under each horizon, and selects the horizon with the highest expected tokens per second.
50%UnverifiedAMTP formulates horizon selection as an online control problem, learning the optimal horizon online through verification feedback.
50%UnverifiedThe paper theoretically proves that AMTP's prediction error translates into throughput regret relative to the best horizon for each context, and that this regret is sublinear when online gradient descent is used.
50%UnverifiedAcross five draft-target pairs built from Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, AMTP achieves up to 4.1x speedup over standard decoding.
50%UnverifiedIn evaluations on AIME 2025 and GSM8K, AMTP matches or exceeds the fixed-window method in 8 of 10 settings, with throughput gains of up to 10%.
50%UnverifiedAMTP keeps the verifier unchanged and is compatible with existing exact verification pipelines, lowering the deployment barrier.
50%