DDP
A distributed training parallelism strategy in which each rank holds an independent model partition across the entire gradient accumulation window and synchronizes gradients only once per optimizer step, combined with a distributed optimizer that shards master weights and optimizer states.
Timeline (last 90 days)
The Olmo-core 3 implementation switches from FSDP to DDP: each rank holds a model partition across the entire accumulation window, gradients are synchronized only once per optimizer step, and the distributed optimizer shards the FP32 master weights and optimizer states; expert parallelism distributes routed experts across ranks, writing only the selected token rows directly into the target rank's buffer via NVSHMEM.
DDP distributes a complete model replica to each GPU and synchronizes gradients via All-Reduce, making it suitable for scenarios where the model fits within a single GPU's memory
Ray Train provides wrappers for both DDP and FSDP strategies, and can automatically handle process group initialization, fault recovery, and checkpoint saving
PyTorch provides native distributed training solutions such as DDP (DistributedDataParallel) and FSDP (Fully Sharded Data Parallel).
AppleTalk internally includes multiple sub-protocols such as AARP (address resolution), DDP (datagram delivery), RTMP (routing), AFP (file sharing), and PAP (printing services).
Compared to DDP, FSDP is more aggressive in overlapping communication with computation, but introduces higher debugging complexity.
All Facts (6)
PyTorch provides native distributed training solutions such as DDP (DistributedDataParallel) and FSDP (Fully Sharded Data Parallel).
60%UnverifiedThe Olmo-core 3 implementation switches from FSDP to DDP: each rank holds a model partition across the entire accumulation window, gradients are synchronized only once per optimizer step, and the distributed optimizer shards the FP32 master weights and optimizer states; expert parallelism distributes routed experts across ranks, writing only the selected token rows directly into the target rank's buffer via NVSHMEM.
50%UnverifiedDDP distributes a complete model replica to each GPU and synchronizes gradients via All-Reduce, making it suitable for scenarios where the model fits within a single GPU's memory
50%UnverifiedRay Train provides wrappers for both DDP and FSDP strategies, and can automatically handle process group initialization, fault recovery, and checkpoint saving
50%UnverifiedAppleTalk internally includes multiple sub-protocols such as AARP (address resolution), DDP (datagram delivery), RTMP (routing), AFP (file sharing), and PAP (printing services).
50%UnverifiedCompared to DDP, FSDP is more aggressive in overlapping communication with computation, but introduces higher debugging complexity.
50%