Unverified50% confidenceFactExact time
In traditional MoE training, a capacity limit (capacity factor) is set for each expert; once an expert receives more tokens than its capacity, the excess tokens are dropped.
1
Sources
50%
Confidence
Long-term
Relevance
9/14/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedMoE training typically requires the introduction of auxiliary loss to enforce load balancing; otherwise, a Matthew effect forms where a minority of Experts become increasingly dominant, causing most Experts to degrade into dead code.74% similarUnverifiedThe core trade-off of MoE is that training requires more GPU memory (all expert weights must be loaded), but FLOPs consumption during inference can be significantly reduced.70% similarVerifiedUnbalanced expert load is the most typical training challenge in MoE, which can lead to the phenomenon of 'expert collapse' and is typically addressed by introducing auxiliary load-balancing loss or token-dropping strategies.69% similarUnverifiedIn the MoE architecture, a routing mechanism dynamically selects the most relevant small subset of experts for each input, typically employing a Top-K strategy (usually K=2), and a load-balancing loss function is introduced during training to prevent uneven load distribution.68% similarUnverifiedMoE training requires an additional load-balancing loss to prevent routing collapse, and DeepSeek-MoE adopts a finer-grained expert partitioning strategy.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/920895API
curl https://kongchang.com/api/v1/knowledge/claims/920895MCP
get_claim(id=920895)