Fine-Grained Experts + Shared Experts: The Dilemma of Routing Exploration in MoE

When MoE Architecture Tries to Have It Both Ways

Mixture-of-Experts (MoE) models have become a core path for scaling up the parameter count of large language models. From early sparse gating designs, to recent fine-grained expert segmentation, and then to the introduction of shared experts, the direction of architectural evolution seems clear: give the model stronger expressive capability at a fixed activation ratio.

But a paper titled Rethinking Routing Exploration in Modern Mixture-of-Experts points out a neglected contradiction — when these two design choices are combined, they may undermine each other.

Two Designs, One Contradiction

To understand this contradiction, we first need to break down what each mechanism does on its own.

Fine-grained expert segmentation splits each expert into smaller units. Under a fixed total activated parameter count, it allows each token to combine more experts. For example, instead of choosing 2 out of 8 experts, the model now chooses 8 out of 64 experts — the combination space explodes from C(8,2)=28 to C(64,8)≈4.4 billion. The routing space expands combinatorially, and the model theoretically gains stronger expressive capability.

Shared experts follow a different route: a subset of experts is activated for all tokens and does not participate in routing selection. Their role is to carry general knowledge, lighten the "generality burden" on routed experts, and let routed experts focus more on differentiated features.

Viewed separately, both designs are reasonable. But the paper finds that when combined, they produce an "exploration dilemma."

Sparse gating and activated parameter count are the foundation for understanding MoE scaling logic. A traditional dense Transformer activates all parameters in each forward pass; an MoE layer, by contrast, uses a gating network (router) to select only a few experts for each token, so the model can have a total parameter count far larger than the compute used in a single pass. The key here is the separation between "activated parameter count" and "total parameter count": total parameters determine the upper bound of model capacity, while activated parameters determine the compute cost per token. Fine-grained expert segmentation does not increase the total parameter count; rather, under a fixed activation budget, it changes the granularity and combination pattern of experts, allowing the same budget to cover more and finer expert units. Therefore, its benefits depend heavily on whether the router can make effective choices in a larger combination space.

How Shared Experts "Kill" Exploration

The core of the problem is that shared experts reduce the diversity of token representations.

Shared experts output the same transformation for all tokens, which is equivalent to imposing a "common projection" on token representations in each layer. As this accumulates across layers, the representation differences between tokens are compressed, and the input received by the routing network becomes more homogeneous.

Routing decisions are essentially a classification problem based on token representations. When token representations become similar, the router tends to repeatedly select the same group of experts, and the incentive to explore new expert combinations decreases.

This creates a contradiction: fine-grained segmentation precisely requires more exploration to realize its advantages — because the combination space is so large that if the router only circulates within a very small subspace, the extra experts and combinations are effectively useless. Shared experts weaken exploration, while fine-grained segmentation requires exploration; the two pull against each other.

The paper's core judgment is: the additional expressive capability provided by fine-grained segmentation may not truly be realized under the suppression of shared experts.

MirrorMoE: Shrinking the Exploration Space Rather Than Encouraging Exploration

Facing this dilemma, the paper proposes MirrorMoE.

Intuitively, since the problem is "insufficient exploration," the solution should be to encourage the router to try more new combinations. But MirrorMoE takes the opposite path: rather than letting the router explore inefficiently in a huge space, it actively suppresses unnecessary experts to shrink the exploration space.

Specifically, MirrorMoE performs adaptive suppression for each token — based on the token's own features, it dynamically masks out experts unlikely to be selected, allowing the router to make decisions within a more compact and relevant candidate set.

The cleverness of this idea is that it does not fight shared experts; instead, it accepts the premise of "homogenized token representations" and then specifically contracts the routing space. The shrinking of the exploration space actually increases the density of effective routing decisions.

The paper reports that MirrorMoE delivers consistent performance improvements across different model scales.

Adaptive expert suppression can be understood as a kind of dynamic candidate-set pruning. Unlike statically reducing the number of experts, MirrorMoE predicts before routing which experts are unlikely to be useful for a given token based on its representation features, and sets those experts' logits or probabilities to extremely low values. In this way, the number of candidate experts the router actually faces decreases, and the decision space contracts from an originally huge combination set to a subset more relevant to the current token. The cost is that useful experts may be mistakenly masked, so the suppression strategy needs to be sufficiently lightweight and learnable. The benefit comes from increased decision density: in a smaller candidate set, the router can more easily distinguish differences among experts and avoid wasting probability mass on many irrelevant experts.

Implications for MoE Architecture Design

What makes this study noteworthy is not the specific method MirrorMoE itself, but the more general problem it reveals: the interaction effects between architectural components are often more important than the independent effects of individual components.

Shared experts and fine-grained segmentation are each validated effective designs, but their combination is not simply "1+1≥2." When multiple design choices are stacked in the same model, unexpected interference may arise. This suggests that architecture search and design cannot rely only on item-by-item ablation experiments; they also need to pay attention to the coupling relationships between components.

Another point worth considering is the concretization of "exploration vs exploitation" in MoE routing. Traditional MoE load-balancing losses try to make expert usage more uniform, but this is more of an engineering technique driven by training stability. MirrorMoE re-examines this problem from an information-theoretic perspective: the router's exploration behavior is constrained by token representation diversity, and token representation diversity is in turn affected by architectural design — this is a deeper causal chain than load balancing.

For researchers and engineers designing or tuning MoE architectures, this paper provides a valuable diagnostic perspective: when you find that fine-grained segmentation does not bring the expected benefits, perhaps the problem is not segmentation itself, but the implicit suppression of routing exploration by shared experts.

Background Supplement

The input to the routing network is the token representation, and token representations undergo multiple residual updates and normalizations in deep networks. Shared experts apply the same or nearly the same transformation to all tokens, which is equivalent to injecting a common component unrelated to token content at each layer. This common component itself helps stabilize training and carry general knowledge, but it increases the similarity between different token representations. When representation similarity rises, the decision boundary for the router's representation-based classification becomes blurred, and multiple tokens are more easily mapped to the same group of experts. As a result, expert utilization decreases, routing diversity contracts, and the "multi-expert combination exploration" that fine-grained segmentation originally relies on is weakened. This mechanism does not depend on an explicit regularization term; it arises naturally from the architecture.

One-line Summary

Fine-grained segmentation combined with shared experts suppresses routing exploration; MirrorMoE resolves the contradiction by shrinking the exploration space.

Paragraph Summary

The article interprets a study on routing exploration in modern MoE. The study points out that when two popular designs — fine-grained expert segmentation and shared experts — are combined, an "exploration dilemma" arises: fine-grained segmentation causes the expert combination space to expand combinatorially, requiring more router exploration; but shared experts impose a common transformation on all tokens, compressing token representation diversity, homogenizing routing inputs, reducing exploration incentives, and preventing the additional expressive capability from being realized. To address this, the paper proposes MirrorMoE, which does not encourage more exploration but instead adaptively suppresses unnecessary experts based on token features, shrinking the candidate space and increasing the density of effective routing decisions. The article emphasizes that interaction effects between architectural components are often more critical than the independent effects of individual components.