DeepSpeed
An open-source large-scale distributed training optimization framework developed by Microsoft and designed for training large models. It supports multiple parallelism strategies, including data parallelism, tensor parallelism, and pipeline parallelism.
Core Facts
Timeline (last 90 days)
During the CAD phase, the team forked Megatron-LM and Microsoft DeepSpeed and rewrote kernels, laying the foundation for the inference capability behind Cursor's more than 500 million model calls per day.
Presenter Masahiro stated that with these updates, DeepSpeed can rival other state-of-the-art frameworks in training the latest MoE models
DeepSpeed recently updated to add tensor parallelism, sequence parallelism, and expert parallelism capabilities
ZeRO (Zero Redundancy Optimizer) reduces GPU memory usage by partitioning optimizer states, gradients, and parameters
PyTorch's torch.save/torch.load, Hugging Face Transformers' Trainer class, DeepSpeed, and Megatron-LM all have built-in periodic checkpoint support.
Large model training involves multiple stages including data cleaning, annotation, alignment, and safety fine-tuning
The numerical inconsistency problem is especially pronounced in heterogeneous architectures that use vLLM or TensorRT-LLM for rollout generation and PyTorch/DeepSpeed for training.
DeepSpeed's ZeRO technology has three stages: Stage 1 shards optimizer states (saving approximately 4× memory), Stage 2 additionally shards gradients (saving approximately 8×), and Stage 3 also shards model parameters.
Megatron-LM implements a 3D parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism, while DeepSpeed distributes optimizer states across multiple nodes via ZeRO technology.
DeepSpeed推出了ZeRO-Infinity，通过利用NVMe SSD的存储空间进一步扩展了可训练模型的规模上限
2 more timeline events
All Facts (16)
Cube Studio supports distributed acceleration frameworks such as DeepSpeed, ColossalAI, and Horovod.
80%UnverifiedZeRO was proposed by the DeepSpeed team in 2020 and eliminates redundancy in data parallelism through three stages: ZeRO-1 shards optimizer states, ZeRO-2 shards gradients, and ZeRO-3 shards model parameters.
70%UnverifiedDeepSpeed and Megatron-LM are mainstream large model training frameworks.
60%UnverifiedMainstream training frameworks such as Megatron-LM and DeepSpeed generate detailed checkpoints and data-loading records during distributed training.
60%UnverifiedTransformers与DeepSpeed、Accelerate等分布式训练工具无缝集成
60%UnverifiedDuring the CAD phase, the team forked Megatron-LM and Microsoft DeepSpeed and rewrote kernels, laying the foundation for the inference capability behind Cursor's more than 500 million model calls per day.
50%UnverifiedPresenter Masahiro stated that with these updates, DeepSpeed can rival other state-of-the-art frameworks in training the latest MoE models
50%UnverifiedZeRO (Zero Redundancy Optimizer) reduces GPU memory usage by partitioning optimizer states, gradients, and parameters
50%UnverifiedDeepSpeed recently updated to add tensor parallelism, sequence parallelism, and expert parallelism capabilities
50%UnverifiedPyTorch's torch.save/torch.load, Hugging Face Transformers' Trainer class, DeepSpeed, and Megatron-LM all have built-in periodic checkpoint support.
50%UnverifiedLarge model training involves multiple stages including data cleaning, annotation, alignment, and safety fine-tuning
50%UnverifiedThe numerical inconsistency problem is especially pronounced in heterogeneous architectures that use vLLM or TensorRT-LLM for rollout generation and PyTorch/DeepSpeed for training.
50%UnverifiedDeepSpeed's ZeRO technology has three stages: Stage 1 shards optimizer states (saving approximately 4× memory), Stage 2 additionally shards gradients (saving approximately 8×), and Stage 3 also shards model parameters.
50%UnverifiedMegatron-LM implements a 3D parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism, while DeepSpeed distributes optimizer states across multiple nodes via ZeRO technology.
50%UnverifiedDeepSpeed推出了ZeRO-Infinity，通过利用NVMe SSD的存储空间进一步扩展了可训练模型的规模上限
50%Unverified微软的 DeepSpeed 库与 ORT Training 有深度集成，两者结合可在千卡规模集群上训练百亿参数级别模型
50%