MTP
An inference acceleration technique that predicts multiple future tokens in a single forward pass and uses a verification mechanism to determine which predictions are accepted, substantially increasing token generation speed. The approach is conceptually related to Speculative Decoding.
Core Facts
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
Speculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
Multi-Token Prediction (MTP) is a training and inference acceleration technique proposed by Meta in 2024 and applied in models such as DeepSeek-V3.
Timeline (last 90 days)
将MTP草稿深度从默认4提升至6或8时，Qwen3.8-27B的输出速度分别降至53.9和52.4 tokens/秒，低于默认的55.1 tokens/秒。
约束解码通过语法掩码保证输出符合JSON等文法，但会破坏MTP、投机解码等并行验证加速路径，导致速度下降，需在可解析性与吞吐速度之间权衡。
在oMLX中启用JSON schema约束解码后，Qwen3.8-27B的30个回答中0个无法解析，但语法约束与MTP无法同时运行，请求退回普通解码，速度降至约25 tokens/秒。
Qwen3.8-27B是稠密模型，全部270亿参数参与每个token的计算，依靠MTP头预测后续token并在单次前向中验证；Ornith 1.5采用MoE架构，每token仅激活约30亿参数，配合DFlash由小型草稿模型提议token块再由主模型校验。
The new version combines D-Flash2, MTP, and N-gram speculative decoding, and the combination of all three produces a roughly twofold decoding acceleration effect.
Strata integrates MTP speculative decoding to improve generation speed, with its decoder weights weighing approximately 0.71 GB (Q2 two-bit quantization).
The number of MTP draft tokens involves a trade-off between acceptance rate and computational overhead; setting it too high may be counterproductive if draft tokens are frequently rejected.
Strata uses the MTP layer for speculative decoding, reporting an average of 2.4 to 3.2 tokens accepted per verification.
The model has a core of 125B parameters, plus an n-gram lookup table with 51B parameters and a multi-token prediction (MTP) draft layer with 4B parameters, totaling approximately 180B.
With a 5070 12 GB GPU paired with 64 GB of memory and MTP enabled, generation speed can exceed 50 tokens/s, with official tests reaching 60–95 tokens/s.
40 more timeline events
All Facts (20)
The MTP (Multi-Token Prediction) auxiliary training objective allows the model to be used with speculative decoding during inference to improve throughput
90%VerifiedSpeculative Decoding uses a small, fast draft model to predict multiple tokens in advance, which are then verified in parallel all at once by the large model.
90%VerifiedMulti-Token Prediction (MTP) is a training and inference acceleration technique proposed by Meta in 2024 and applied in models such as DeepSeek-V3.
85%VerifiedMTP is a technique that allows the model to predict multiple subsequent tokens in a single forward pass, reducing the number of inference computation rounds.
80%VerifiedMTP adds multiple prediction heads to the model's output layer, enabling the model to simultaneously predict the probability distribution of the next N tokens in a single forward pass, achieving 2–4x inference acceleration without sacrificing generation quality.
80%VerifiedDeepSeek's MTP base-version draft model is co-optimized during main model training to align the draft model's output distribution with the target model, distinguishing it from post-hoc distillation approaches.
75%VerifiedDeepSeek introduced MTP technology as early as version V3.
75%VerifiedCompared to speculative decoding, MTP's predictive capability is built into the model itself, requiring no additional draft model.
65%VerifiedMTP (Multi-Token Prediction) is a configuration flag that is disabled by default in most mainstream inference tools and can accelerate inference.
65%VerifiedMTP (Multi-Token Prediction) technology was systematically proposed by Meta in 2024 research and has been adopted by mainstream open-source models such as Qwen and DeepSeek.
65%UnverifiedMTP一次前向传播可同时产出多个token：主头输出token+1，第一个MTP头输出token+2，第二个MTP头输出token+3
90%UnverifiedMTP的额外预测头直接复用Transformer的隐藏状态表示，因此额外的计算开销极小
90%UnverifiedMTP（多令牌预测）最初由Meta在研究论文《Better & Faster Large Language Models via Multi-token Prediction》中系统提出
85%UnverifiedMTP与DFlash在架构上有本质不同，但都属于投机解码技术谱系
85%Unverified加载MTP层后额外浮点运算会导致数值产生微小变化，可能导致模型走不同推理路径产生不同结果，即使温度设为0也会出现
85%UnverifiedGoogle在Gemma 4中实际应用了MTP技术
80%UnverifiedMeta在LLaMA 3的技术报告中首次将MTP作为训练目标引入
80%UnverifiedMTP和Speculative Decoding两项技术理论上可以叠加使用
75%UnverifiedThrough MTP (Multi-Token Prediction) technology, the inference speed of Qwen3.8-Flash-Next and GLM-5.3-Flash is improved by up to 2x.
70%UnverifiedThe Qwen3 series natively supports MTP.
60%