Muon
A deep learning optimizer designed by Jordan Juravsky and collaborators specifically for matrix parameters. Its key innovation is orthogonalizing gradient matrices after momentum updates, based on Nesterov momentum and Newton-Schulz iteration. It has demonstrated faster convergence than AdamW in model training efficiency competitions.
Timeline (last 90 days)
The Muon optimizer was designed by Jordan Juravsky et al. for matrix parameters; its key innovation is applying orthogonalization to the gradient matrix after the momentum update.
In the NanoGPT Speedrun, Muon is typically applied only to the matrix weights of hidden layers, while the embedding layer and output layer retain AdamW.
The decoder and lookup table use the AdamW optimizer (learning rate 1×10⁻⁵), while the recurrent weights use the Muon optimizer (learning rate 0.005, momentum 0.95).
The Muon optimizer, proposed by Jeremy Bernstein et al., is centered on the idea of orthogonalizing gradient matrices via Newton-Schulz iteration.
All Facts (5)
Muon优化器由Kosson等研究者提出，核心改进在于对梯度更新矩阵进行正交化处理
85%UnverifiedThe Muon optimizer was designed by Jordan Juravsky et al. for matrix parameters; its key innovation is applying orthogonalization to the gradient matrix after the momentum update.
50%UnverifiedIn the NanoGPT Speedrun, Muon is typically applied only to the matrix weights of hidden layers, while the embedding layer and output layer retain AdamW.
50%UnverifiedThe decoder and lookup table use the AdamW optimizer (learning rate 1×10⁻⁵), while the recurrent weights use the Muon optimizer (learning rate 0.005, momentum 0.95).
50%UnverifiedThe Muon optimizer, proposed by Jeremy Bernstein et al., is centered on the idea of orthogonalizing gradient matrices via Newton-Schulz iteration.
50%