Spherical Optimization
A training method that constrains the direction of each row of a weight matrix to the unit sphere, equivalent to row-wise normalization; it can be regarded as a degenerate form of the orthogonality constraint.
Timeline (last 90 days)
When d=1, the orthogonal constraint degenerates to spherical optimization, i.e., the direction of each row lies on its own unit sphere, which by itself already outperforms the tuned Muon optimizer.
The researchers also conducted experiments on diffusion Transformers trained on ImageNet and obtained the same two findings: block orthogonality brings no additional benefit, and spherical optimization is already sufficient.
For practitioners, if the goal is to improve pretraining efficiency, prioritizing spherical optimization may be the simpler and more effective choice.
All Facts (3)
When d=1, the orthogonal constraint degenerates to spherical optimization, i.e., the direction of each row lies on its own unit sphere, which by itself already outperforms the tuned Muon optimizer.
50%UnverifiedThe researchers also conducted experiments on diffusion Transformers trained on ImageNet and obtained the same two findings: block orthogonality brings no additional benefit, and spherical optimization is already sufficient.
50%UnverifiedFor practitioners, if the goal is to improve pretraining efficiency, prioritizing spherical optimization may be the simpler and more effective choice.
50%