Unverified50% confidenceFactExact time
In large-scale distributed training, forward propagation, backward propagation, and AllReduce/AllGather communication operations are interleaved; if a communication operation is not complete, downstream compute operators must wait, creating bubbles.
1
Sources
50%
Confidence
Long-term
Relevance
10/6/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedCommunication primitives in distributed training (such as AllReduce gradient aggregation) can introduce non-deterministic accumulated errors due to differences in parallelism configuration and network topology.69% similarUnverifiedTensor parallelism splits a single-layer weight matrix along a certain dimension, and requires one All-Reduce communication at the end of each forward pass step, making communication frequency extremely high.63% similarUnverified大语言模型多卡张量并行时，每一层前向传播都需要在卡间同步激活值，带宽瓶颈会直接拉低整体推理吞吐63% similarUnverified分布式拆层运行超大模型方案中设备间传输中间状态的网络延迟是主要瓶颈，更适合对延迟不敏感的批处理任务而非实时对话60% similarUnverifiedIn multi-agent pipelines, cross-model communication loss accumulates and amplifies as the number of nodes increases.59% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/982167API
curl https://kongchang.com/api/v1/knowledge/claims/982167MCP
get_claim(id=982167)