Block Orthogonality
A constraint scheme in which a weight matrix is divided into blocks of d rows each, with each block forming a row-orthogonal Stiefel matrix; the block size d is a hyperparameter.
Timeline (last 90 days)
Since a strictly orthogonal square matrix loses half its degrees of freedom, the researchers studied block orthogonality, i.e., every d rows form a row-orthogonal (Stiefel) matrix, with the block size d as a hyperparameter.
Experiments show that over block sizes ranging from 1 to the shorter side of the matrix, no setting with d>1 reliably outperforms d=1 across the entire network, and large blocks are clearly worse.
The researchers also conducted experiments on diffusion Transformers trained on ImageNet and obtained the same two findings: block orthogonality brings no additional benefit, and spherical optimization is already sufficient.
All Facts (3)
Since a strictly orthogonal square matrix loses half its degrees of freedom, the researchers studied block orthogonality, i.e., every d rows form a row-orthogonal (Stiefel) matrix, with the block size d as a hyperparameter.
50%UnverifiedExperiments show that over block sizes ranging from 1 to the shorter side of the matrix, no setting with d>1 reliably outperforms d=1 across the entire network, and large blocks are clearly worse.
50%UnverifiedThe researchers also conducted experiments on diffusion Transformers trained on ImageNet and obtained the same two findings: block orthogonality brings no additional benefit, and spherical optimization is already sufficient.
50%