Token Choice
A routing strategy in MoE whereby each token selects the top-k experts from all experts to process it. It guarantees token coverage but cannot control expert load.
Timeline (last 90 days)
In Token Choice routing, each token selects the top-k experts with the highest scores from N experts for processing, which naturally guarantees that each token is processed by at least one expert, but it cannot control the number of tokens each expert receives.
In an experiment with a 32-expert miniature character model, when Token Choice routing had no load balancing loss, a single expert swallowed 57% of the tokens in the early stage of training, and 17 of the 32 experts were almost ignored.
All Facts (2)
In Token Choice routing, each token selects the top-k experts with the highest scores from N experts for processing, which naturally guarantees that each token is processed by at least one expert, but it cannot control the number of tokens each expert receives.
50%UnverifiedIn an experiment with a 32-expert miniature character model, when Token Choice routing had no load balancing loss, a single expert swallowed 57% of the tokens in the early stage of training, and 17 of the 32 experts were almost ignored.
50%