MuonGS
An LLM pretraining optimization method based on an improved Muon optimizer that removes the radial component of updates through Gram-Schmidt projection, relaxing weight-norm control to accelerate training.
Timeline (last 90 days)
The paper proposes an optimizer method called MuonGS that accelerates LLM pretraining by relaxing weight norm control.
MuonGS uses Gram-Schmidt projection to remove the radial component in each Muon update without fixing the weight norm.
Theoretical analysis shows that under MuonGS the weight norm grows by at most a factor of √t after t steps, guaranteeing a cumulative effective learning rate C_T = Ω(√T).
For smooth scale-invariant objectives, MuonGS establishes a stationarity guarantee of O(T^{-1/2}).
Under long training cycles, MuonGS outperforms MuonH on validation perplexity.
On dense models, compared with MuonH, MuonGS is estimated to achieve about a 15% token-equivalent speedup when matching validation loss.
The design of MuonGS both retains the benefits of weight-norm control and avoids the limitations imposed by hard constraints.
All Facts (7)
The paper proposes an optimizer method called MuonGS that accelerates LLM pretraining by relaxing weight norm control.
50%UnverifiedMuonGS uses Gram-Schmidt projection to remove the radial component in each Muon update without fixing the weight norm.
50%UnverifiedTheoretical analysis shows that under MuonGS the weight norm grows by at most a factor of √t after t steps, guaranteeing a cumulative effective learning rate C_T = Ω(√T).
50%UnverifiedFor smooth scale-invariant objectives, MuonGS establishes a stationarity guarantee of O(T^{-1/2}).
50%UnverifiedUnder long training cycles, MuonGS outperforms MuonH on validation perplexity.
50%UnverifiedOn dense models, compared with MuonH, MuonGS is estimated to achieve about a 15% token-equivalent speedup when matching validation loss.
50%UnverifiedThe design of MuonGS both retains the benefits of weight-norm control and avoids the limitations imposed by hard constraints.
50%