FSDP2
PyTorch's fully sharded data parallel training scheme (Fully Sharded Data Parallel v2), which shards model parameters, gradients, and optimizer states across multiple GPUs to reduce per-GPU memory usage.
Timeline (last 90 days)
FSDP2 maintains parameters and gradients as per-parameter tensors, while its collective communication path operates on flat rank-major buffers.
In Llama-3.1 70B training (8K tokens per GPU), layout conversion alone accounts for 6.4% of step time.
Unsloth supports distributing LoRA, QLoRA and full fine-tuning across multiple GPUs in Python using FSDP2, and can split long sequences via context_parallel_size.
Version 5.18.0 unified device_mesh initialization to support pipeline-parallel plus tensor-parallel inference, and combined FSDP2 with expert parallelism via a two-dimensional (fsdp, tp) device mesh.
All Facts (4)
FSDP2 maintains parameters and gradients as per-parameter tensors, while its collective communication path operates on flat rank-major buffers.
50%UnverifiedIn Llama-3.1 70B training (8K tokens per GPU), layout conversion alone accounts for 6.4% of step time.
50%UnverifiedUnsloth supports distributing LoRA, QLoRA and full fine-tuning across multiple GPUs in Python using FSDP2, and can split long sequences via context_parallel_size.
50%UnverifiedVersion 5.18.0 unified device_mesh initialization to support pipeline-parallel plus tensor-parallel inference, and combined FSDP2 with expert parallelism via a two-dimensional (fsdp, tp) device mesh.
50%