Fine-Grained Expert Segmentation
A design in MoE architectures in which each expert is split into smaller units, increasing the number of experts that a token can be combined with while keeping the total number of activated parameters unchanged, thereby enlarging the routing combination space.
Timeline (last 90 days)
Fine-grained expert partitioning, while keeping the total number of activated parameters unchanged, inflates the expert selection combination space from, for example, C(8,2)=28 to C(64,8)≈4.4 billion.
The paper "Rethinking Routing Exploration in Modern Mixture-of-Experts" points out that fine-grained expert partitioning combined with shared experts produces an exploration dilemma, weakening each other.
The paper's core judgment holds that the extra expressive capacity provided by fine-grained partitioning may not actually be realized under the suppression of shared experts.
All Facts (3)
Fine-grained expert partitioning, while keeping the total number of activated parameters unchanged, inflates the expert selection combination space from, for example, C(8,2)=28 to C(64,8)≈4.4 billion.
50%UnverifiedThe paper "Rethinking Routing Exploration in Modern Mixture-of-Experts" points out that fine-grained expert partitioning combined with shared experts produces an exploration dilemma, weakening each other.
50%UnverifiedThe paper's core judgment holds that the extra expressive capacity provided by fine-grained partitioning may not actually be realized under the suppression of shared experts.
50%