Token-Choice Routing
The most common routing form in LLM MoE architectures. The routing network produces scores over all experts for each token, and the top-K experts are selected to process that token. It is simple to implement and supports variable-length inputs.
Timeline (last 90 days)
PixelMoE directly adopts the standard Token-Choice routing from LLMs, keeps one shared expert, and uses a bias-based load balancing strategy, without designing any diffusion-specific routing mechanism or auxiliary routing objective function.
In Token-Choice routing, the routing network produces scores for all experts for each token, then selects the Top-K experts to process that token; its inherent risk is imbalanced expert utilization, and common countermeasures include additive bias to adjust routing scores, auxiliary load balancing losses, and expert capacity limits.
All Facts (2)
PixelMoE directly adopts the standard Token-Choice routing from LLMs, keeps one shared expert, and uses a bias-based load balancing strategy, without designing any diffusion-specific routing mechanism or auxiliary routing objective function.
50%UnverifiedIn Token-Choice routing, the routing network produces scores for all experts for each token, then selects the Top-K experts to process that token; its inherent risk is imbalanced expert utilization, and common countermeasures include additive bias to adjust routing scores, auxiliary load balancing losses, and expert capacity limits.
50%