Muon-RT
An improved optimizer method for self-distillation training of large language models. It replaces the momentum buffer by reusing the parameter displacement between the EMA teacher and the student, thereby reducing persistent memory usage while maintaining convergence. The method targets the memory overhead associated with maintaining separate momentum state during self-distillation.
Timeline (last 90 days)
Muon-RT replaces an independent momentum buffer by reusing the parameter displacement between the EMA teacher and the student model, thereby eliminating the momentum buffer for hidden weight matrices and achieving more memory-efficient self-distillation training.
Muon-RT's processing pipeline is: remove the radial component from the teacher-student displacement, perform spectral normalization, and then mix it with the current gradient after Newton-Schulz orthogonalization.
Muon-RT's design remains compatible with Muon's original Newton-Schulz orthogonalization pipeline; the changes are concentrated in the source of the momentum term rather than the overall optimization framework.
Under explicit regularity and alignment assumptions, the Muon-RT paper establishes a convergence result in which the expected objective gap bound contracts linearly toward the residual error level.
Muon-RT's convergence guarantee relies on an alignment assumption, namely that the inner product between the teacher displacement and the current gradient is non-negative; if the teacher lags too long and deviates severely from the student's current direction, the assumption may be violated.
Under the SDFT setting on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy by 2.6 percentage points over AdamW while reducing persistent memory by 42%.
Under the SDFT setting on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy by 5.4 percentage points over Muon with momentum while reducing persistent memory by 27%.
Under the SDPO setting on Olmo-3-7B-Instruct, Muon-RT improves average accuracy by 2.0 percentage points over AdamW.
Muon-RT's experiments are currently concentrated on 7B-scale models, and its performance at larger scales remains to be seen.
All Facts (9)
Muon-RT replaces an independent momentum buffer by reusing the parameter displacement between the EMA teacher and the student model, thereby eliminating the momentum buffer for hidden weight matrices and achieving more memory-efficient self-distillation training.
50%UnverifiedMuon-RT's processing pipeline is: remove the radial component from the teacher-student displacement, perform spectral normalization, and then mix it with the current gradient after Newton-Schulz orthogonalization.
50%UnverifiedMuon-RT's design remains compatible with Muon's original Newton-Schulz orthogonalization pipeline; the changes are concentrated in the source of the momentum term rather than the overall optimization framework.
50%UnverifiedUnder explicit regularity and alignment assumptions, the Muon-RT paper establishes a convergence result in which the expected objective gap bound contracts linearly toward the residual error level.
50%UnverifiedMuon-RT's convergence guarantee relies on an alignment assumption, namely that the inner product between the teacher displacement and the current gradient is non-negative; if the teacher lags too long and deviates severely from the student's current direction, the assumption may be violated.
50%UnverifiedUnder the SDFT setting on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy by 2.6 percentage points over AdamW while reducing persistent memory by 42%.
50%UnverifiedUnder the SDFT setting on Qwen2.5-7B-Instruct, Muon-RT improves average accuracy by 5.4 percentage points over Muon with momentum while reducing persistent memory by 27%.
50%UnverifiedUnder the SDPO setting on Olmo-3-7B-Instruct, Muon-RT improves average accuracy by 2.0 percentage points over AdamW.
50%UnverifiedMuon-RT's experiments are currently concentrated on 7B-scale models, and its performance at larger scales remains to be seen.
50%