Verified65% confidenceFactExact time
In MoE implementations, without constraints, the router tends to concentrate token assignments to a small number of experts, leading to Expert Collapse.
3
Sources
65%
Confidence
Long-term
Relevance
7/13/2026
First Seen
Sources
15岁少年从零训练200M MoE语言模型：完整架构与实战拆解
redditr/learnmachinelearning7/12/2026
Related Claims
UnverifiedExpert collapse refers to the phenomenon where the routing network tends to route most tokens to a small number of experts, leaving the remaining experts idle, which can be mitigated by penalizing imbalanced routing with an auxiliary loss function.77% similarUnverifiedMixture-of-Experts (MoE) models use a routing mechanism to select only a small number of experts for each inference pass, reducing per-inference computation, but the knowledge of the unselected experts cannot be invoked.76% similarUnverified负载均衡损失是MoE实现中的关键挑战，若不加干预会导致专家坍塌（Expert Collapse）现象75% similarUnverifiedMoE稀疏激活特性使推理时实际计算量仅与激活的专家数量挂钩，而非总参数量72% similarUnverifiedThe load-balancing problem of MoE routers requires mechanisms such as auxiliary loss functions to force balanced expert utilization, but this may conflict with the primary task objective.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/502526API
curl https://kongchang.com/api/v1/knowledge/claims/502526MCP
get_claim(id=502526)