Expert Choice
A routing strategy in MoE whereby each expert selects a fixed number of top-k tokens from all tokens to process. It guarantees balanced expert load but may result in tokens being dropped.
Timeline (last 90 days)
In Expert Choice routing, each expert selects a fixed number of top-k tokens from all tokens according to its own preference; load balancing on the expert dimension is strictly guaranteed mathematically, but there may be cases where some tokens are not selected by any expert.
In three rounds of controlled experiments on a 32-expert tiny character model, Expert Choice routing achieved the lowest evaluation loss among the three rounds.
The community is exploring hybrid strategies, such as adding token coverage constraints on top of Expert Choice, or using a learnable routing temperature to dynamically adjust the degree of balance.
All Facts (3)
In Expert Choice routing, each expert selects a fixed number of top-k tokens from all tokens according to its own preference; load balancing on the expert dimension is strictly guaranteed mathematically, but there may be cases where some tokens are not selected by any expert.
50%UnverifiedIn three rounds of controlled experiments on a 32-expert tiny character model, Expert Choice routing achieved the lowest evaluation loss among the three rounds.
50%UnverifiedThe community is exploring hybrid strategies, such as adding token coverage constraints on top of Expert Choice, or using a learnable routing temperature to dynamically adjust the degree of balance.
50%