[KongchangAI]
ConceptCollective Communication Primitives

Communication Primitives

Collective communication operations used for data exchange between cards in multi-GPU training and inference, such as AllReduce, AllGather, Broadcast, and ReduceScatter. They are typically implemented by collective communication libraries such as NCCL and form the basis for implementing parallelism strategies.

Timeline (last 90 days)

Nov 2

In multi-GPU training and inference, inter-GPU data exchange relies on communication primitives such as AllReduce, AllGather, Broadcast, ReduceScatter, etc., which are usually implemented by collective communication libraries such as NCCL.

Unverified50%
Nov 2

Different parallelism strategies (data parallelism, tensor parallelism, pipeline parallelism) have different requirements for the combinations and frequency of communication primitives, and communication patterns are an important dimension of workload classification.

Unverified50%
Nov 2

If a new framework's parallelism strategy cannot be mapped to existing communication primitives, multi-GPU performance will degrade significantly.

Unverified50%

All Facts (3)

Source Articles