Unverified50% confidenceFactExact time
The most common form of load balancing loss is the auxiliary loss, represented by Switch Transformer, with the formula L_balance = α · N · Σ(f_i · p_i), where N is the number of experts and α is the loss coefficient.
1
Sources
50%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Understanding MoE Routing: Three Core Problems and Two Fixes
redditr/deeplearning10/15/2026
Related Entities
Related Claims
UnverifiedThe load balancing loss uses the sum of squares of all expert selection frequencies L=ΣFi², and by the Cauchy–Schwarz inequality the loss is minimized when all Fi=1/N.74% similarUnverifiedThe weight update formula in AdaBoost derives from the step-by-step minimization of the exponential loss function L=Σ_i e^(-y_i F(x_i)).72% similarUnverifiedGradient descent computes the partial derivatives of the loss function with respect to model parameters and updates the parameters in the direction opposite to the gradient to approach the minimum of the loss function.68% similarUnverifiedThe coefficient α of the load balancing loss is usually taken to be around 0.01, but the specific value is highly dependent on model scale and data distribution.68% similarUnverified绝大多数常见损失函数都可以从最大似然估计推导出来68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1035488API
curl https://kongchang.com/api/v1/knowledge/claims/1035488MCP
get_claim(id=1035488)