Stable rank
The ratio of the squared Frobenius norm to the squared spectral norm of a matrix; it reflects the degree of energy concentration and is a continuous rank measure sensitive to singular value decay.
Timeline (last 90 days)
Expert outputs in MoE layers exhibit structural redundancy, manifested as the stable rank and entropy effective rank of expert outputs being far lower than the nominal number of experts, and the output distributions having fewer effective modes.
Stable rank is the ratio of the squared Frobenius norm to the squared spectral norm of a matrix, i.e., ||W||_F^2 / ||W||_2^2; it does not exceed the nominal rank of the matrix and is sensitive to singular value decay.
All Facts (2)
Expert outputs in MoE layers exhibit structural redundancy, manifested as the stable rank and entropy effective rank of expert outputs being far lower than the nominal number of experts, and the output distributions having fewer effective modes.
50%UnverifiedStable rank is the ratio of the squared Frobenius norm to the squared spectral norm of a matrix, i.e., ||W||_F^2 / ||W||_2^2; it does not exceed the nominal rank of the matrix and is sensitive to singular value decay.
50%