Multi-View MoE
A method for improving the routing quality of sparse mixture-of-experts (MoE) models. It aggregates the preferences of multiple independently initialized routers and reuses routing preferences in the expert inputs, improving expert selection and computation without increasing the activated computation.
Timeline (last 90 days)
Multi-View MoE improves expert selection by aggregating the sigmoid scores of multiple independently initialized routers, expanding the diversity of candidate signals without increasing the number of activated experts.
Multi-View MoE further exploits each selected expert's relative scores across different routers, mixing routing preferences into the expert input so that they are used for both selection and computation.
After training a 24-layer, 10B-A1B model on 708B tokens, the full Multi-View MoE improves the average score on nine downstream tasks by 1.18 percentage points over the baseline.
Multi-View MoE yields a global relative improvement of 8.1% in the rank correlation between router scores and expert utility, and a relative improvement of about 58% in the layer with the lowest single-router alignment.
Multi-View MoE does not increase inference cost: each selected expert still performs only one forward pass, and the number of activated experts, expert FLOPs, and all-to-all communication volume all remain consistent with the baseline.
Multi-View MoE has so far been validated only at the 10B-A1B scale, and whether the gains hold for larger models and larger numbers of routers remains to be seen.
Controlled mixed-intervention experiments show that the input differences exploited by the model depend on both expert and token factors, and routing preferences are not static labels but dynamically coupled with the specific input.
All Facts (7)
Multi-View MoE improves expert selection by aggregating the sigmoid scores of multiple independently initialized routers, expanding the diversity of candidate signals without increasing the number of activated experts.
50%UnverifiedMulti-View MoE further exploits each selected expert's relative scores across different routers, mixing routing preferences into the expert input so that they are used for both selection and computation.
50%UnverifiedAfter training a 24-layer, 10B-A1B model on 708B tokens, the full Multi-View MoE improves the average score on nine downstream tasks by 1.18 percentage points over the baseline.
50%UnverifiedMulti-View MoE yields a global relative improvement of 8.1% in the rank correlation between router scores and expert utility, and a relative improvement of about 58% in the layer with the lowest single-router alignment.
50%UnverifiedMulti-View MoE does not increase inference cost: each selected expert still performs only one forward pass, and the number of activated experts, expert FLOPs, and all-to-all communication volume all remain consistent with the baseline.
50%UnverifiedMulti-View MoE has so far been validated only at the 10B-A1B scale, and whether the gains hold for larger models and larger numbers of routers remains to be seen.
50%UnverifiedControlled mixed-intervention experiments show that the input differences exploited by the model depend on both expert and token factors, and routing preferences are not static labels but dynamically coupled with the specific input.
50%