Adaptive Multi-Token Prediction: How Does the Speculative Horizon Dynamically Adjust for Inference Acceleration?

Title

Adaptive Multi-Token Prediction: How Does the Speculative Horizon Dynamically Adjust for Inference Acceleration?

Meta Description

This article explains how Adaptive Multi-Token Prediction (AMTP) frames speculative horizon selection as an online control problem, using verification feedback to dynamically adjust draft length, achieving up to 4.1x decoding speedup on models like Llama and Qwen, with theoretical guarantees and experimental analysis.

Article Content

Multi-Token Prediction (MTP) can significantly accelerate language model decoding by drafting and verifying multiple tokens in a single target model forward pass. But a key tension remains: the optimal speculative horizon varies with context. Drafting longer amortizes cost better when reliability is high, but wasted computation follows if drafts are rejected. Unlike autoregressive drafting, which can stop token by token, MTP must fix the horizon before drafting because all heads fire in the same forward pass. How to dynamically select the horizon becomes the core problem for improving MTP's practical efficiency.

Why a Fixed Horizon Is Not Optimal

MTP's acceleration principle is "draft-and-verify": lightweight heads predict multiple future tokens at once, then the target model verifies them in parallel. If drafted tokens are accepted, decoding steps decrease; if rejected, the drafting computation is wasted. Longer horizons offer greater potential gains but higher rejection risk. The original paper points out that the optimal horizon varies with context—some contexts draft reliably and suit long horizons, while others fail easily and are more economical with short horizons. A fixed horizon cannot adapt to this variation, leading to wasted computation or insufficient acceleration in some scenarios.

Speculative Decoding is the background framework for understanding MTP's acceleration principle. Standard autoregressive decoding generates only one token per step, requiring a full call to the large model each time, resulting in high latency. Speculative decoding introduces a lightweight draft model (or draft head) to quickly generate multiple candidate tokens, then the target model verifies these candidates in parallel in a single forward pass: if verification passes, multiple tokens are accepted at once; otherwise, it rolls back to the first rejected position. Its speedup comes from the verification stage processing multiple candidates in parallel, while drafting cost is far lower than the target model. MTP is essentially a variant of speculative decoding, differing in that draft heads share the backbone with the target model or are built into the model, and all heads fire in the same forward pass, so it cannot stop dynamically token by token like an independent draft model and must fix the draft horizon in advance.

AMTP: Modeling Horizon Selection as an Online Control Problem

The paper proposes Adaptive Multi-Token Prediction (AMTP), formalizing horizon selection as an online control problem. The core is to use verification feedback to learn how trustworthy drafting is under the current context. AMTP is a lightweight controller that requires no model retraining and does not change the exact verifier. It characterizes the expected number of committed tokens at each horizon through "acceptance survival"—the probability that a drafted prefix is fully accepted. Before each round, AMTP predicts how far the draft is likely to survive and selects the horizon with the highest expected tokens per second, learning for free from each verification.

Online Control refers to a class of sequential decision-making frameworks in which the decision-maker adjusts strategy step by step based on historical feedback without knowing the future environment in advance. Unlike optimal control in classical control theory, where the system model is known, it emphasizes learning while deciding. AMTP treats horizon selection before each drafting round as an online decision: the controller observes current context features, predicts expected returns under different horizons, selects one, then obtains feedback from verification results and updates. Since the context distribution is unknown and varies with input, the globally optimal horizon cannot be solved offline; the online control framework allows the system to continuously approach the optimal policy for each context during decoding—this is precisely the theoretical basis for "no need to know the context in advance."

Theoretical Guarantees: Prediction Error Converts into Throughput Regret

The paper theoretically proves that any online learner's prediction error converts into throughput regret relative to the best horizon for each context, which in turn converts into a bound on wall-clock completion time. With online gradient descent, this regret is sublinear. This means that as decoding proceeds, AMTP can gradually approach the optimal horizon policy without knowing the context in advance.

Regret is a core metric in online learning, referring to the difference between the cumulative return of an online policy and the cumulative return of "the best decision in hindsight at each step." If regret grows as o(T) with the number of steps T—that is, average regret tends to zero—the policy is said to have sublinear regret, meaning that over the long run the online policy can approach the optimal fixed policy. The throughput regret in this paper defines each step's return as tokens committed per second, measuring AMTP's throughput loss relative to the best horizon for each context. Algorithms such as online gradient descent can guarantee O(√T) regret under convex or approximately convex assumptions, thereby converting prediction error into an upper bound on wall-clock completion time and providing formal grounding for AMTP's convergence.

Experimental Results: Up to 4.1x Speedup, Outperforming Fixed Windows in Most Scenarios

Across five draft-target pairs built from Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP model, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP achieves up to 4.1x speedup over standard decoding, matches or exceeds fixed windows in 8 of 10 settings, and delivers throughput gains of up to 10%. These results show that dynamic horizon selection can further unlock MTP's acceleration potential without sacrificing verification accuracy.

Implications for Inference Acceleration Practice

AMTP's value lies in turning "how far to draft" from a hyperparameter into an online-learnable decision. For teams deploying MTP, this means no manual horizon tuning for different tasks—the system adapts based on real-time verification feedback. At the same time, AMTP keeps the verifier unchanged, remaining compatible with existing exact verification pipelines and lowering the barrier to adoption. In the future, this online control approach could extend to other speculative decoding variants, enabling acceleration strategies to match context more intelligently.

One-line Summary

AMTP models MTP draft horizon selection as online control, dynamically adapting to context without retraining, with up to 4.1x speedup.

Paragraph Summary

Multi-Token Prediction (MTP) accelerates decoding by drafting and verifying multiple tokens in a single forward pass, but its optimal speculative horizon varies with context, and a fixed horizon leads to wasted computation or insufficient acceleration. The paper proposes Adaptive Multi-Token Prediction (AMTP), formalizing horizon selection as an online control problem, using verification feedback to learn the acceptance survival probability of drafted prefixes, and selecting the horizon with the highest expected tokens per second each round—without retraining the model or changing the exact verifier. Theory proves that prediction error converts into sublinear throughput regret relative to the best per-context horizon. Experiments on five pairs including Llama-3.1-8B-Instruct, Meta MTP models, and Qwen3-4B show that AMTP achieves up to 4.1x speedup over standard decoding, matches or exceeds fixed windows in 8 of 10 settings, and delivers throughput gains of up to 10%.