Noise Residual

The noise residual is the remaining part of the MTP gradient that is orthogonal to the KL self-distillation signal. It does not correspond to a beneficial learning objective and interferes with the standard next-token prediction loss. It may originate from the accumulated error of multi-step prediction and from gradient conflicts.