Expert Parallelism (EP)

A parallelism strategy commonly used in mixture-of-experts (MoE) models that distributes different expert subnetworks across different devices. Each token is sent to the corresponding expert device for computation according to routing results. It can scale model capacity, but the communication pattern is complex.