Expert Parallelism
A partitioning approach for distributed inference of Mixture-of-Experts (MoE) models. Expert networks are spread across different GPUs, with each GPU hosting only a subset of experts, reducing per-GPU memory pressure.
Timeline (last 90 days)
MoE models in distributed training typically require the simultaneous deployment of four strategies: data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism.
Expert parallelism places different experts on different devices, using All-to-All communication to dynamically route and distribute tokens.
Expert Parallelism assigns different experts to different GPU devices, and the routing and aggregation of tokens across devices requires efficient All-to-All communication primitives.
Expert Parallelism places high demands on interconnect bandwidth (such as NVLink and InfiniBand).
In multi-node multi-GPU deployments, expert parallelism introduces All-to-All communication across devices, which is one of the main reasons MoE has higher latency than dense models of equivalent compute in distributed environments.
All Facts (5)
MoE models in distributed training typically require the simultaneous deployment of four strategies: data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism.
50%UnverifiedExpert parallelism places different experts on different devices, using All-to-All communication to dynamically route and distribute tokens.
50%UnverifiedExpert Parallelism assigns different experts to different GPU devices, and the routing and aggregation of tokens across devices requires efficient All-to-All communication primitives.
50%UnverifiedExpert Parallelism places high demands on interconnect bandwidth (such as NVLink and InfiniBand).
50%UnverifiedIn multi-node multi-GPU deployments, expert parallelism introduces All-to-All communication across devices, which is one of the main reasons MoE has higher latency than dense models of equivalent compute in distributed environments.
50%