Does Orthogonality Actually Help LLM Pretraining? A Large-Scale Empirical Study Says No

The Orthogonality Hype and the Pretraining Gap

In recent years, a large body of work has reported benefits from using orthogonality constraints during the fine-tuning stage of large language models (LLMs). These results naturally raise a question: does orthogonality also help LLM pretraining? According to a systematic empirical study, researchers pretrained decoder-only LLMs ranging from 360M to 3.1B parameters using nearly 200B tokens, examined fourteen orthogonal training variants, and ultimately concluded that, in the implementations they tried, row-to-row orthogonality provides no additional benefit for language model pretraining beyond sphere optimization.

This conclusion conflicts with multiple prior works that reported orthogonality improvements, prompting the researchers to re-examine five methods that had previously reported orthogonality gains in pretraining and fine-tuning. They found that in each case, the reported gains could be attributed to numerical precision, training duration, optimizer implementation, or per-row norm control, rather than to the orthogonality imposed by the method itself.

Fourteen Variants and an Exploration of Block Orthogonality

The study covered fourteen orthogonal training variants: some directly constrain the weight matrices themselves, while others reparameterize the weight matrices using orthogonal factors. Because a strictly orthogonal square matrix loses half of its degrees of freedom, the researchers also studied block-wise orthogonality, where every (d) rows form a row-orthogonal (Stiefel) matrix, with the block size (d) as a hyperparameter.

Experiments show that, across block sizes from 1 to the shorter side of the matrix, no setting with (d>1) reliably outperforms (d=1) across the entire network, and large blocks are clearly worse. The only helpful placement—using a symmetric positive-definite factor on the query and key projections—has a tiny gain that vanishes as the model grows.

When (d=1), the constraint degenerates into sphere optimization, where each row's direction lies on its own unit sphere, which already outperforms a tuned Muon optimizer.

Cross-Validation: Diffusion Transformers and Fine-Tuning Results

To test the generality of the conclusion, the researchers also ran experiments on diffusion Transformers trained on ImageNet, obtaining the same two findings: block orthogonality provides no additional benefit, and sphere optimization is already sufficient. Moreover, after fine-tuning, downstream accuracy cannot distinguish between different parameterizations.

These results are consistent across multiple model architectures and tasks, strengthening the credibility of the conclusion that "orthogonality provides no additional benefit."

Re-examining Prior Work: Where Do the Gains Come From?

Since the conclusion conflicts with multiple prior works, the researchers carefully re-inspected five methods that reported orthogonality improvements. The analysis shows that the gains reported by these methods actually stem from:

• Numerical precision: computational differences under different precisions may be mistaken for the credit of orthogonality constraints.
• Training duration: improvements from longer training time are misattributed.
• Optimizer implementation: differences in the internal implementation details of optimizers lead to different results.
• Per-row norm control: control over each row's norm, rather than orthogonality itself, brings the improvement.

In other words, the "benefits" of orthogonality constraints have not been rigorously validated in the pretraining setting.

Implications for LLM Training Practice

This study reminds us to be cautious when introducing orthogonality constraints into LLM pretraining. Sphere optimization (i.e., row normalization with (d=1)) already provides better results than the Muon optimizer, while more complex orthogonality constraints bring no additional benefit and may instead increase computational overhead. For practitioners, if the goal is to improve pretraining efficiency, prioritizing sphere optimization may be a simpler and more effective choice.

At the same time, the study emphasizes that when reporting empirical gains, variables must be strictly controlled and the effects of factors such as numerical precision and training duration must be distinguished, to avoid misattributing irrelevant factors to the method itself.

Conclusion

This large-scale empirical study systematically examined the role of orthogonality constraints in LLM pretraining, covering training from 360M to 3.1B parameters with nearly 200B tokens, as well as cross-validation on diffusion Transformers. The results show that row orthogonality provides no additional help for pretraining beyond sphere optimization, and that previously reported gains can be attributed to other factors. This finding has important reference value for the design and evaluation of LLM training methods.

One-line Summary

Large-scale empirical evidence shows that row orthogonality provides no additional gain in LLM pretraining beyond sphere optimization, and previously reported benefits can be attributed to other factors.

Paragraph Summary

A systematic empirical study pretrained decoder-only LLMs from 360M to 3.1B parameters using nearly 200B tokens and examined fourteen orthogonal training variants. Results show that block orthogonality (d>1) does not reliably outperform sphere optimization (d=1), with larger blocks performing worse; sphere optimization itself already outperforms a tuned Muon optimizer. Cross-validation on diffusion Transformers yields the same conclusion, and downstream accuracy after fine-tuning cannot distinguish between different parameterizations. The researchers re-examined five methods that previously reported orthogonality improvements and found that the gains can be attributed to numerical precision, training duration, optimizer implementation, or per-row norm control, rather than orthogonality itself. This reminds practitioners to introduce orthogonality constraints cautiously, prioritize sphere optimization, and strictly control variables when reporting gains.