All-to-All
A collective communication operation in which each device distributes data to all other devices and receives data from them. In MoE expert parallelism it is used to route tokens to the corresponding experts; its communication volume is sensitive to bandwidth and latency.
Timeline (last 90 days)
All-to-All is a collective communication operation in which each device distributes the data it holds to all other devices while receiving data from all other devices.
TorchTitan's fine-grained control can schedule and overlap All-to-All communication, hiding communication behind computation and thereby alleviating the problem of insufficient interconnect bandwidth on public clusters.
In MoE expert parallelism, every forward and backward pass requires All-to-All communication, whose volume is related to the number of tokens, the number of experts and the hidden dimension.
Expert parallelism distributes different experts across different GPUs and uses All-to-All communication to dispatch and gather tokens, which is key to implementing MoE training systems.
Unlike AllReduce, the traffic pattern of All-to-All is highly irregular and difficult to predict, and load-balancing imbalances can create hotspots.
All-to-All communication requires each card to send different data blocks to every other card individually, with communication volume growing quadratically with the number of cards.
Expert Parallelism assigns different experts to different GPU devices, and the routing and aggregation of tokens across devices requires efficient All-to-All communication primitives.
All Facts (7)
All-to-All is a collective communication operation in which each device distributes the data it holds to all other devices while receiving data from all other devices.
50%UnverifiedTorchTitan's fine-grained control can schedule and overlap All-to-All communication, hiding communication behind computation and thereby alleviating the problem of insufficient interconnect bandwidth on public clusters.
50%UnverifiedIn MoE expert parallelism, every forward and backward pass requires All-to-All communication, whose volume is related to the number of tokens, the number of experts and the hidden dimension.
50%UnverifiedExpert parallelism distributes different experts across different GPUs and uses All-to-All communication to dispatch and gather tokens, which is key to implementing MoE training systems.
50%UnverifiedUnlike AllReduce, the traffic pattern of All-to-All is highly irregular and difficult to predict, and load-balancing imbalances can create hotspots.
50%UnverifiedAll-to-All communication requires each card to send different data blocks to every other card individually, with communication volume growing quadratically with the number of cards.
50%UnverifiedExpert Parallelism assigns different experts to different GPU devices, and the routing and aggregation of tokens across devices requires efficient All-to-All communication primitives.
50%