Concept
NVSHMEM
A PGAS communication library from NVIDIA based on the OpenSHMEM specification, supporting direct memory access and efficient communication between GPUs.
Timeline (last 90 days)
Oct 14
The Olmo-core 3 implementation switches from FSDP to DDP: each rank holds a model partition across the entire accumulation window, gradients are synchronized only once per optimizer step, and the distributed optimizer shards the FP32 master weights and optimizer states; expert parallelism distributes routed experts across ranks, writing only the selected token rows directly into the target rank's buffer via NVSHMEM.
Unverified50%