All-to-All Communication

A collective communication primitive in distributed computing in which every node sends data to and receives data from every other node. It is the core communication pattern for token routing in Mixture-of-Experts (MoE) models. All-to-all operations are associated with high communication overhead and irregular traffic patterns, posing challenges for scalable distributed training.