Multi-View MoE: Letting Each Router Reshape Expert Computation

Sparse Mixture-of-Experts (MoE) models scale parameter capacity by activating only a small number of experts per token, and the quality of expert selection directly determines model capability under a fixed activation budget. The conventional approach relies on a single router to score and rank experts, but a new study points out a clear misalignment between this ranking and the actual utility of experts, and proposes Multi-View MoE, which uses multiple routers and router-preference reuse to simultaneously improve selection and computation.

The Limits of a Single Router: The Selected Experts May Not Be the Most Useful

By densely evaluating all experts, the research team found that the consistency between the ranking of a single router and the ranking from expert utility evaluation is weak, especially in the middle and deep layers. This means that the experts the router "favors" are not necessarily the ones that actually improve model performance.

Obtaining richer selection signals is not cheap. Selection based on internal expert activations requires a precomputation step first; and feeding multiple full representations separately into the same expert duplicates that expert's computation. Under a fixed expert budget, both paths erode the efficiency advantage of MoE.

The basic mechanism of Mixture-of-Experts (MoE): An MoE layer consists of a set of parallel feed-forward networks (i.e., "experts") and a router. For each token, the router outputs scores for each expert, typically taking the Top-K (e.g., Top-2) experts with the highest scores, and only the outputs of these experts are weighted and summed for the return pass. As a result, the total parameter count can far exceed the number of parameters actually involved in computation per token, forming a sparse structure with "large parameter capacity, small activated computation." The so-called "fixed activation budget" means that the number of experts activated per token, K, and the expert hidden dimension are fixed, so expert FLOPs and all-to-all communication volume are essentially constant. It is precisely under this budget constraint that the router's "choice of whom to select" becomes the key variable determining model quality—no matter how many experts there are, choosing the wrong ones amounts to wasting the capacity dividend brought by sparsity.

Rank correlation: A statistic measuring the consistency of two rankings, commonly the Spearman rank correlation coefficient, ranging from -1 to 1; the closer to 1, the more consistent the relative order of the same set of experts under the two scoring schemes. The paper uses it to compare "the ranking by router scores" with "the ranking by utility obtained from dense per-expert evaluation." If rank correlation is low, it indicates that router preference is decoupled from actual gains. Note that rank correlation reflects only order consistency, not the calibration of absolute score values, so it can serve as a diagnostic metric for routing quality but cannot be directly equated with model accuracy.

The Two Core Designs of Multi-View MoE

Multi-Router: Aggregating Independently Initialized Preferences

The Multi-Router submodule of Multi-View MoE aggregates sigmoid scores from multiple independently initialized routers. During training, these routers develop highly differentiated expert preferences, so that experts strongly preferred by one router, or jointly supported by multiple routers, can both enter the selection range. This mechanism essentially expands the diversity of candidate signals without increasing the number of activated experts.

Router-Preference Reuse: Letting Selection Signals Participate in Computation

Better selection alone is not enough. Multi-View further exploits the relative scores of each selected expert across different routers, mixing the learned latent views into that expert's input, thereby using routing preference for both selection and computation. In other words, the router not only decides "who plays" but also influences "how to play."

The key is that efficiency is not sacrificed: each selected expert still performs only one forward pass, and the number of activated experts, expert FLOPs, and all-to-all communication volume all remain unchanged.

The difference between Sigmoid routing and Softmax routing: Traditional MoE mostly uses Softmax routers, where expert scores are normalized into probabilities, naturally creating competition among experts with scores summing to 1. Sigmoid routing scores each expert independently without cross-expert normalization, allowing "multiple experts to be highly preferred at the same time," and is also more convenient for introducing load-balancing auxiliary losses during training. Multi-View MoE adopts sigmoid scores precisely so that multiple independently initialized routers can each form distinct and non-mutually-exclusive preference distributions; during aggregation, the scores of the routers are then combined, so that both signals—"strongly preferred by a certain router" and "jointly supported by multiple routers"—are incorporated into the candidates. Relative scores are the proportional relationships obtained by comparing expert scores under the same token, which can be regarded as a lightweight "view" weight used to modulate expert inputs with almost no additional computation.

Experimental Data: Average Improvement of 1.18 Percentage Points Across Nine Tasks

On a 24-layer, 10B-A1B model trained with 708B tokens, the full Multi-View MoE improved the average score across nine downstream tasks by 1.18 percentage points over the baseline. Using only the Multi-Router submodule already brings a gain of 0.6 percentage points.

More noteworthy is the improvement in rank correlation between router scores and evaluated expert utility: a global relative improvement of 8.1%, and in the layers where single-router alignment is lowest, a relative improvement of about 58%. This shows that the multi-router mechanism fills the gap precisely where a single router "fails" most.

Controlled mixture intervention experiments further show that the input differences exploited by the model depend on both expert and token factors—routing preference is not a static label but dynamically coupled with specific inputs.

The Balance Point Between Efficiency and Effectiveness

The idea of Multi-View MoE offers a general insight for MoE architecture design: improving the quality of expert selection does not necessarily require extra computation. By reusing routing preference, selection signals and computation signals are unified into the same mechanism, avoiding the overhead of precomputation or repeated forward passes.

For scenarios that need to squeeze model capability under a fixed activation budget, this "zero extra FLOPs" improvement path has direct engineering value. However, the paper has so far validated only at the 10B-A1B scale, and whether the gains hold for larger models and more routers remains to be seen.

Frequently Asked Questions

Will Multi-View MoE increase inference cost? No. Each selected expert still performs only one forward pass, and the number of activated experts, expert FLOPs, and all-to-all communication volume all remain consistent with the baseline.

What is the difference between multi-router and traditional expert selection? Traditional methods rely on the ranking of a single router. Multi-router aggregates the sigmoid scores of multiple independently initialized routers, using their differentiated preferences to expand the candidate range and improve the correlation between router scores and expert utility.

In which layers is the gain most pronounced? The gains are more prominent in the middle and deep layers, especially in the layers where single-router alignment is lowest, where rank correlation improves by about 58% relatively.

One-line Summary

Multi-View MoE aggregates preferences with multiple routers and reuses routing signals to improve expert selection quality with zero extra FLOPs.

Paragraph Summary

This article introduces a study on the routing mechanism of sparse Mixture-of-Experts (MoE) models. The study points out that the consistency between the score ranking of a traditional single router and the actual utility of experts is weak, especially in the middle and deep layers; moreover, the cost of obtaining richer selection signals through precomputation or repeated forward passes erodes the efficiency advantage of MoE. To address this, the authors propose Multi-View MoE, which includes two core designs: Multi-Router aggregates the sigmoid scores of multiple independently initialized routers, using differentiated expert preferences to expand candidate signals; router-preference reuse mixes the relative scores of each selected expert across different routers into its input, so that routing signals participate in both selection and computation. After training a 24-layer, 10B-A1B model with 708B tokens, the full method improves the average score across nine downstream tasks by 1.18 percentage points, with Multi-Router alone improving it by 0.6 percentage points; the rank correlation between router scores and expert utility improves by 8.1% globally in relative terms, and by about 58% relatively in the layers with the lowest alignment, while the number of activated experts, expert FLOPs, and all-to-all communication volume all remain unchanged.