Unverified50% confidenceFactExact time
The coefficient α of the load balancing loss is usually taken to be around 0.01, but the specific value is highly dependent on model scale and data distribution.
1
Sources
50%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Understanding MoE Routing: Three Core Problems and Two Fixes
redditr/deeplearning10/15/2026
Related Entities
Related Claims
UnverifiedThe load balancing loss uses the sum of squares of all expert selection frequencies L=ΣFi², and by the Cauchy–Schwarz inequality the loss is minimized when all Fi=1/N.75% similarUnverifiedWith well-calibrated FP8 inference, the loss in model output quality can typically be kept within 1%.73% similarUnverified不同归一化方案对同一模型的IC影响可达0.01-0.03的量级69% similarUnverifiedWhen the quantile level τ=0.5, quantile loss degenerates to Mean Absolute Error (MAE).69% similarUnverifiedFocal Loss adds a modulating factor (1-p_t)^γ before the standard cross-entropy loss, with γ typically set to 2, causing the model to focus on hard examples.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1035489API
curl https://kongchang.com/api/v1/knowledge/claims/1035489MCP
get_claim(id=1035489)