Shared Expert
A design in MoE architectures in which a subset of experts is activated for all tokens and does not participate in routing selection, intended to carry general knowledge and reduce the general-purpose burden on routed experts.
Timeline (last 90 days)
Shared experts output the same transformation for all tokens, equivalent to applying a common projection to token representations at each layer; accumulated over multiple layers, this compresses the differences between token representations, making the routing network's input more homogeneous.
The paper "Rethinking Routing Exploration in Modern Mixture-of-Experts" points out that fine-grained expert partitioning combined with shared experts produces an exploration dilemma, weakening each other.
The paper's core judgment holds that the extra expressive capacity provided by fine-grained partitioning may not actually be realized under the suppression of shared experts.
The common component produced by shared experts helps stabilize training and carry general knowledge, but increases the similarity between different token representations.
Shared experts are activated every time and do not participate in Top-K selection; they are used to extract the general foundational capabilities required by all tokens.
DeepSeek-V2 adopted a configuration of 1 shared expert plus multiple routed experts, achieving results superior to pure routed MoE.
All Facts (6)
Shared experts output the same transformation for all tokens, equivalent to applying a common projection to token representations at each layer; accumulated over multiple layers, this compresses the differences between token representations, making the routing network's input more homogeneous.
50%UnverifiedThe paper "Rethinking Routing Exploration in Modern Mixture-of-Experts" points out that fine-grained expert partitioning combined with shared experts produces an exploration dilemma, weakening each other.
50%UnverifiedThe paper's core judgment holds that the extra expressive capacity provided by fine-grained partitioning may not actually be realized under the suppression of shared experts.
50%UnverifiedThe common component produced by shared experts helps stabilize training and carry general knowledge, but increases the similarity between different token representations.
50%UnverifiedShared experts are activated every time and do not participate in Top-K selection; they are used to extract the general foundational capabilities required by all tokens.
50%UnverifiedDeepSeek-V2 adopted a configuration of 1 shared expert plus multiple routed experts, achieving results superior to pure routed MoE.
50%