Isotropic Curvature Model
A theoretical model explaining the effectiveness of the Muon optimizer. Under an isotropic curvature assumption, it approximates the local loss as a curvature model and proves that gradient orthogonalization minimizes a one-step local loss surrogate.
Timeline (last 90 days)
The Isotropic Curvature Model (ICM) explains the effectiveness of Muon from the perspective of curvature: when gradient orthogonalization minimizes a one-step local loss surrogate, Muon's update direction is reasonable.
ICM assumes that the input radius is fixed, but in real training the radius of the input activations of each layer varies with the data distribution and is not constant.
Under common continuous radial distributions, the mean-curvature interpretation of ICM can hardly provide non-trivial support for Muon's update, and Muon's optimality becomes extremely rare.
All Facts (3)
The Isotropic Curvature Model (ICM) explains the effectiveness of Muon from the perspective of curvature: when gradient orthogonalization minimizes a one-step local loss surrogate, Muon's update direction is reasonable.
50%UnverifiedICM assumes that the input radius is fixed, but in real training the radius of the input activations of each layer varies with the data distribution and is not constant.
50%UnverifiedUnder common continuous radial distributions, the mean-curvature interpretation of ICM can hardly provide non-trivial support for Muon's update, and Muon's optimality becomes extremely rare.
50%