[KongchangAI]
Concept全对全通信

All-to-All

A collective communication operation in which each device distributes data to all other devices and receives data from them. In MoE expert parallelism it is used to route tokens to the corresponding experts; its communication volume is sensitive to bandwidth and latency.

Timeline (last 90 days)

Nov 12

All-to-All is a collective communication operation in which each device distributes the data it holds to all other devices while receiving data from all other devices.

Unverified50%
Nov 12

TorchTitan's fine-grained control can schedule and overlap All-to-All communication, hiding communication behind computation and thereby alleviating the problem of insufficient interconnect bandwidth on public clusters.

Unverified50%
Nov 12

In MoE expert parallelism, every forward and backward pass requires All-to-All communication, whose volume is related to the number of tokens, the number of experts and the hidden dimension.

Unverified50%
Nov 5

Expert parallelism distributes different experts across different GPUs and uses All-to-All communication to dispatch and gather tokens, which is key to implementing MoE training systems.

Unverified50%
Oct 4

Unlike AllReduce, the traffic pattern of All-to-All is highly irregular and difficult to predict, and load-balancing imbalances can create hotspots.

Unverified50%
Oct 3

All-to-All communication requires each card to send different data blocks to every other card individually, with communication volume growing quadratically with the number of cards.

Unverified50%
Sep 24

Expert Parallelism assigns different experts to different GPU devices, and the routing and aggregation of tokens across devices requires efficient All-to-All communication primitives.

Unverified50%

All Facts (7)

Source Articles