Wide Expert Parallelism
A parallelization deployment strategy for Mixture-of-Experts (MoE) models that spreads experts horizontally across a larger number of devices, with each device hosting independent expert weights. By expanding the device pool, this approach achieves load balancing and is particularly effective for large models with high expert counts.
Timeline (last 90 days)
At the GPU level, with wide expert parallelism (wide-EP), the expert load carried on different GPUs can vary enormously, causing some GPUs to become bottlenecks and slowing the processing of the entire batch.
Wide expert parallelism (wide-EP) means splitting experts across more GPUs (i.e., a large EP scale) so that each GPU carries fewer experts, which can reduce per-card memory pressure but amplifies the risk of load imbalance.
All Facts (2)
At the GPU level, with wide expert parallelism (wide-EP), the expert load carried on different GPUs can vary enormously, causing some GPUs to become bottlenecks and slowing the processing of the entire batch.
50%UnverifiedWide expert parallelism (wide-EP) means splitting experts across more GPUs (i.e., a large EP scale) so that each GPU carries fewer experts, which can reduce per-card memory pressure but amplifies the risk of load imbalance.
50%