all-reduce
All-reduce is a collective communication primitive that aggregates partial results distributed across multiple GPUs into a complete result and broadcasts it back to all participants. It is a key communication operation in each layer's forward computation under tensor parallelism, highly sensitive to bandwidth and latency, and with ring algorithms its time cost is roughly proportional to the number of participating GPUs.
Timeline (last 90 days)
Typical scenarios of abstraction leakage include distributed training getting stuck at all-reduce, mixed precision causing gradient overflow, and custom operators not being supported.
张量并行每一层前向计算几乎都要执行一次all-reduce等集合通信，通信频率随层数线性增长，对带宽和延迟极度敏感。
all-reduce在环形算法下，其耗时约与参与卡数成正比。
In collective communication operations such as all-reduce, a network anomaly on a single GPU is sufficient to slow down the entire training job.
In data-parallel training, each GPU independently computes mini-batch gradients, then synchronizes them back to all cards via all-reduce by summing or averaging the gradients across all GPUs to keep model parameters consistent.
All Facts (5)
Typical scenarios of abstraction leakage include distributed training getting stuck at all-reduce, mixed precision causing gradient overflow, and custom operators not being supported.
50%Unverified张量并行每一层前向计算几乎都要执行一次all-reduce等集合通信，通信频率随层数线性增长，对带宽和延迟极度敏感。
50%Unverifiedall-reduce在环形算法下，其耗时约与参与卡数成正比。
50%UnverifiedIn collective communication operations such as all-reduce, a network anomaly on a single GPU is sufficient to slow down the entire training job.
50%UnverifiedIn data-parallel training, each GPU independently computes mini-batch gradients, then synchronizes them back to all cards via all-reduce by summing or averaging the gradients across all GPUs to keep model parameters consistent.
50%