Soft-MTP
A method that improves Multi-Token Prediction (MTP) training by modifying the residual connection of hidden states (h_ntp += h_mtp.detach() - h_mtp), decoupling the KL self-distillation signal from noisy residuals so that gradients flow only through the beneficial KL self-distillation component, thereby improving cross-task performance without additional computational overhead.
Timeline (last 90 days)
Soft-MTP can be implemented with one line of code, `h_ntp += h_mtp.detach() - h_mtp`, by modifying the residual connection of the hidden state.
Soft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
Soft-MTP adds no forward or backward computation, introduces no new parameters, and does not change the model structure at inference time.
Soft-MTP outperforms standard MTP on a variety of tasks, but the original material did not disclose specific benchmark data.
The effectiveness of Soft-MTP depends on the training quality of the original MTP head; if the MTP head is underfit, the separable KL signal is limited.
The research team has open-sourced Soft-MTP's code, paper, and HuggingFace model collection.
All Facts (6)
Soft-MTP can be implemented with one line of code, `h_ntp += h_mtp.detach() - h_mtp`, by modifying the residual connection of the hidden state.
50%UnverifiedSoft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
50%UnverifiedSoft-MTP adds no forward or backward computation, introduces no new parameters, and does not change the model structure at inference time.
50%UnverifiedSoft-MTP outperforms standard MTP on a variety of tasks, but the original material did not disclose specific benchmark data.
50%UnverifiedThe effectiveness of Soft-MTP depends on the training quality of the original MTP head; if the MTP head is underfit, the separable KL signal is limited.
50%UnverifiedThe research team has open-sourced Soft-MTP's code, paper, and HuggingFace model collection.
50%