AllReduce
A core collective communication operation in distributed deep learning training, used to aggregate and broadcast gradient data across multiple compute nodes. It represents one of the primary network communication bottlenecks in large-scale AI training workloads.
Timeline (last 90 days)
在 batch size = 1 的小消息交换场景下，AllReduce 的性能由延迟主导，而非带宽，因此 x16 相对 x8 并不能显著降低延迟。
AllReduce is essentially distributed floating-point summation, whose result depends on the response order of participating nodes and is inherently nondeterministic in asynchronous network environments.
Tensor parallelism splits a single matrix operation across multiple GPUs working together, requiring an AllReduce communication to aggregate results after each operation.
常见的集合通信操作包括AllReduce、AllGather和ReduceScatter，其中AllReduce将所有节点的梯度求和后广播回每个节点
AllReduce在每个训练迭代反向传播结束后要求所有GPU同步梯度，这一步骤在数千卡规模下可能消耗数十毫秒甚至更长时间
In large-scale distributed training, forward propagation, backward propagation, and AllReduce/AllGather communication operations are interleaved; if a communication operation is not complete, downstream compute operators must wait, creating bubbles.
In collective communication operations during AI training (such as AllReduce gradient aggregation), overall performance is determined by the slowest node.
Communication primitives in distributed training (such as AllReduce gradient aggregation) can introduce non-deterministic accumulated errors due to differences in parallelism configuration and network topology.
Unlike AllReduce, the traffic pattern of All-to-All is highly irregular and difficult to predict, and load-balancing imbalances can create hotspots.
In tensor-parallel inference, AllReduce operations occur at every layer, and for large models with tens or even hundreds of layers, the cumulative volume of communicated data is considerable.
1 more timeline events
All Facts (12)
在千卡规模下，一次AllReduce的延迟可能占到单步训练时间的30%以上
60%Unverified在 batch size = 1 的小消息交换场景下，AllReduce 的性能由延迟主导，而非带宽，因此 x16 相对 x8 并不能显著降低延迟。
50%UnverifiedAllReduce is essentially distributed floating-point summation, whose result depends on the response order of participating nodes and is inherently nondeterministic in asynchronous network environments.
50%UnverifiedTensor parallelism splits a single matrix operation across multiple GPUs working together, requiring an AllReduce communication to aggregate results after each operation.
50%Unverified常见的集合通信操作包括AllReduce、AllGather和ReduceScatter，其中AllReduce将所有节点的梯度求和后广播回每个节点
50%UnverifiedAllReduce在每个训练迭代反向传播结束后要求所有GPU同步梯度，这一步骤在数千卡规模下可能消耗数十毫秒甚至更长时间
50%UnverifiedIn large-scale distributed training, forward propagation, backward propagation, and AllReduce/AllGather communication operations are interleaved; if a communication operation is not complete, downstream compute operators must wait, creating bubbles.
50%UnverifiedIn collective communication operations during AI training (such as AllReduce gradient aggregation), overall performance is determined by the slowest node.
50%UnverifiedCommunication primitives in distributed training (such as AllReduce gradient aggregation) can introduce non-deterministic accumulated errors due to differences in parallelism configuration and network topology.
50%UnverifiedUnlike AllReduce, the traffic pattern of All-to-All is highly irregular and difficult to predict, and load-balancing imbalances can create hotspots.
50%UnverifiedIn tensor-parallel inference, AllReduce operations occur at every layer, and for large models with tens or even hundreds of layers, the cumulative volume of communicated data is considerable.
50%UnverifiedData parallelism shards training data across multiple GPUs, where each GPU holds a complete model replica and independently computes gradients, then synchronizes gradients through collective communication operations such as AllReduce.
50%