Routing Collapse
A phenomenon in MoE training in which a positive feedback loop formed by the cold start of the routing network and expert networks causes a small number of experts to receive the vast majority of tokens while most experts remain insufficiently trained.
Timeline (last 90 days)
The essence of routing collapse is a positive feedback loop: an expert gains more tokens due to a slight advantage at initialization, undergoes gradient updates more frequently, adapts its parameters to the data distribution faster, and thereby obtains a higher probability in subsequent routing decisions.
For teams using MoE architectures, it is recommended to monitor expert load distribution and token coverage early in training; once routing collapse occurs, it is very difficult to fix later by continuing training.
Routing collapse is a classic challenge in training mixture-of-experts and dynamic routing architectures, where routers easily fall into local optima causing all tokens to be assigned to the same path.
All Facts (3)
The essence of routing collapse is a positive feedback loop: an expert gains more tokens due to a slight advantage at initialization, undergoes gradient updates more frequently, adapts its parameters to the data distribution faster, and thereby obtains a higher probability in subsequent routing decisions.
50%UnverifiedFor teams using MoE architectures, it is recommended to monitor expert load distribution and token coverage early in training; once routing collapse occurs, it is very difficult to fix later by continuing training.
50%UnverifiedRouting collapse is a classic challenge in training mixture-of-experts and dynamic routing architectures, where routers easily fall into local optima causing all tokens to be assigned to the same path.
50%