In multi-node multi-GPU deployments, expert parallelism introduces All-to-All communication across devices, which is one of the main reasons MoE has higher latency than dense models of equivalent compute in distributed environments.