Load Balancing Loss
A regularization technique commonly used in MoE training. By adding a penalty term to the training objective, it encourages tokens to be distributed evenly across experts in order to curb routing collapse.
Timeline (last 90 days)
The traditional MoE load balancing loss mainly starts from training stability, making expert usage more uniform.
The most common form of load balancing loss is the auxiliary loss, represented by Switch Transformer, with the formula L_balance = α · N · Σ(f_i · p_i), where N is the number of experts and α is the loss coefficient.
The coefficient α of the load balancing loss is usually taken to be around 0.01, but the specific value is highly dependent on model scale and data distribution.
The load balancing loss introduces a constraint unrelated to language modeling, and there is tension between this constraint and the main task objective, so tuning the loss coefficient requires experience.
The load balancing loss uses the sum of squares of all expert selection frequencies L=ΣFi², and by the Cauchy–Schwarz inequality the loss is minimized when all Fi=1/N.
All Facts (5)
The traditional MoE load balancing loss mainly starts from training stability, making expert usage more uniform.
50%UnverifiedThe most common form of load balancing loss is the auxiliary loss, represented by Switch Transformer, with the formula L_balance = α · N · Σ(f_i · p_i), where N is the number of experts and α is the loss coefficient.
50%UnverifiedThe coefficient α of the load balancing loss is usually taken to be around 0.01, but the specific value is highly dependent on model scale and data distribution.
50%UnverifiedThe load balancing loss introduces a constraint unrelated to language modeling, and there is tension between this constraint and the main task objective, so tuning the loss coefficient requires experience.
50%UnverifiedThe load balancing loss uses the sum of squares of all expert selection frequencies L=ΣFi², and by the Cauchy–Schwarz inequality the loss is minimized when all Fi=1/N.
50%