Unverified50% confidenceFactExact time
In data-parallel training, BatchNorm's mean and variance need to be aggregated via all-reduce across all devices, and each BN layer triggers cross-device communication in both the forward and backward passes.
1
Sources
50%
Confidence
Long-term
Relevance
11/9/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe ideas of splitting sequential communication into parallelizable levels and decoupling cross-hardware from intra-hardware communication also have reference value for other parallel strategies that rely on frequent collective communication.69% similarUnverifiedParallel topology should match the hardware communication structure, constraining high-frequency communication within groups rather than having all GPUs participate in global synchronization.68% similarUnverifiedTensor parallelism splits the weight matrix of the same layer and places it across multiple machines, requiring one All-Reduce communication per layer computed, is extremely sensitive to bandwidth and latency, and is usually only practical under high-speed interconnects such as NVLink.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1151073API
curl https://kongchang.com/api/v1/knowledge/claims/1151073MCP
get_claim(id=1151073)