Entropy effective rank
An effective dimensionality measure based on the Shannon entropy of normalized singular values; complementary to stable rank, it characterizes the degree of structured redundancy in a matrix's output.
Timeline (last 90 days)
Expert outputs in MoE layers exhibit structural redundancy, manifested as the stable rank and entropy effective rank of expert outputs being far lower than the nominal number of experts, and the output distributions having fewer effective modes.
Entropy effective rank is based on the Shannon entropy of normalized singular values: normalize the squared singular values into a probability distribution, compute its entropy, and take the exponential to obtain the effective number of dimensions.
All Facts (2)
Expert outputs in MoE layers exhibit structural redundancy, manifested as the stable rank and entropy effective rank of expert outputs being far lower than the nominal number of experts, and the output distributions having fewer effective modes.
50%UnverifiedEntropy effective rank is based on the Shannon entropy of normalized singular values: normalize the squared singular values into a probability distribution, compute its entropy, and take the exponential to obtain the effective number of dimensions.
50%