NCCL
NCCL (NVIDIA Collective Communications Library) is a multi-GPU collective communication library developed by NVIDIA, designed to accelerate inter-GPU communication in deep learning training across multiple GPUs and nodes. It provides standard collective communication primitives including AllReduce, Broadcast, Reduce, and AllGather, and fully utilizes high-speed interconnects such as NVLink and InfiniBand to achieve low-latency, high-bandwidth data exchange between GPUs. NCCL is widely integrated into major deep learning frameworks including TensorFlow and PyTorch, serving as a core communication component for large-scale distributed training.
Core Facts
Timeline (last 90 days)
NCCL（NVIDIA Collective Communications Library）是目前最广泛使用的GPU集合通信库，但其底层仍依赖主机侧的调度逻辑
There is a strict version-binding relationship between InfiniBand/RoCE network drivers (MLNX_OFED) and the NCCL communication library.
DeepEP corresponds to NCCL, handling communication between multiple GPUs; FlashMLA corresponds to FlashAttention, used for ultra-long text processing; TileKernels corresponds to CUTLASS; DeepSelect corresponds to data pipelines such as DALI.
RCCL is typically used as a transparent replacement for NCCL via environment variables or plugin mechanisms, requiring no modification to upper-layer framework code.
RCCL (ROCm Collective Communications Library) is a collective communications library developed by AMD for the ROCm platform, functionally equivalent to NVIDIA's NCCL.
The NVIDIA version of DeepEP uses NCCL, with NVLink within a node, RDMA across nodes, and kernels written in CUDA.
NVIDIA has designed NVLink high-speed interconnects and the NCCL communication library specifically for multi-card training, forming a complete hardware-software solution.
NVIDIA's NCCL is a communication library responsible for data synchronization across multiple cards and multiple machines during training, directly affecting the efficiency of large-scale training
The communication performance of NCCL (NVIDIA Collective Communications Library) is highly dependent on kernel parameter settings of the inter-node network stack, such as socket buffer sizes and RDMA-related configurations.
Daisy demonstrated running the Cosmos 3 video generation model on two DGX Sparks connected via Ethernet, using context parallelism with low-latency NCCL for data communication.
5 more timeline events
All Facts (20)
CUDA是NVIDIA于2006年推出的并行计算平台和编程模型
80%UnverifiedNCCL通信库已经针对NVLink拓扑做了深度优化
90%UnverifiedNCCL内部实现了Ring、Tree、Direct等多种集合通信算法以适应不同的网络拓扑和消息大小
90%UnverifiedNCCL Inspector直接嵌入NCCL运行时环境，以极低的性能开销采集每一次集合通信操作的关键指标
85%UnverifiedThe barriers of the CUDA ecosystem are deeply rooted in operator libraries (cuBLAS, cuDNN), compilation optimization toolchains (NVCC, TensorRT), and distributed training frameworks (NCCL), forming NVIDIA's deep moat.
60%UnverifiedNCCL（NVIDIA Collective Communications Library）是目前最广泛使用的GPU集合通信库，但其底层仍依赖主机侧的调度逻辑
50%UnverifiedThere is a strict version-binding relationship between InfiniBand/RoCE network drivers (MLNX_OFED) and the NCCL communication library.
50%UnverifiedDeepEP corresponds to NCCL, handling communication between multiple GPUs; FlashMLA corresponds to FlashAttention, used for ultra-long text processing; TileKernels corresponds to CUTLASS; DeepSelect corresponds to data pipelines such as DALI.
50%UnverifiedRCCL is typically used as a transparent replacement for NCCL via environment variables or plugin mechanisms, requiring no modification to upper-layer framework code.
50%UnverifiedRCCL (ROCm Collective Communications Library) is a collective communications library developed by AMD for the ROCm platform, functionally equivalent to NVIDIA's NCCL.
50%UnverifiedThe NVIDIA version of DeepEP uses NCCL, with NVLink within a node, RDMA across nodes, and kernels written in CUDA.
50%UnverifiedNVIDIA has designed NVLink high-speed interconnects and the NCCL communication library specifically for multi-card training, forming a complete hardware-software solution.
50%UnverifiedNVIDIA's NCCL is a communication library responsible for data synchronization across multiple cards and multiple machines during training, directly affecting the efficiency of large-scale training
50%UnverifiedThe communication performance of NCCL (NVIDIA Collective Communications Library) is highly dependent on kernel parameter settings of the inter-node network stack, such as socket buffer sizes and RDMA-related configurations.
50%UnverifiedThe scheduling strategies of collective communication operations such as AllReduce, AllGather, and ReduceScatter, as well as the version selection of the underlying NCCL/RCCL libraries, can impact end-to-end throughput of 10,000-GPU cluster training by more than 20%.
50%UnverifiedNCCL's communication protocol is designed with greater emphasis on performance than security, and rarely encrypts or performs integrity checks on gradient data transmitted between nodes.
50%UnverifiedNVIDIA的NCCL是目前工业界使用最广泛的AllReduce实现，针对NVLink、PCIe和InfiniBand等互联拓扑做了优化
50%UnverifiedML训练的完整GPU计算栈包括GPU驱动、CUDA Runtime、cuDNN、cuBLAS以及NCCL多GPU集合通信库
50%UnverifiedCUDA生态已积累了cuBLAS、cuDNN、NCCL、TensorRT等数十个专业库
50%UnverifiedNCCL是NVIDIA专为多GPU通信设计的库，支持AllReduce、Broadcast、AllGather等集合通信原语
50%