Megatron-LM
A distributed training framework for large-scale language models developed by NVIDIA, supporting the combination of data parallelism, tensor parallelism, and pipeline parallelism. It is designed for training extremely large models across thousands of GPUs and serves as the underlying training engine of the NeMo framework.
Core Facts
Timeline (last 90 days)
During the CAD phase, the team forked Megatron-LM and Microsoft DeepSpeed and rewrote kernels, laying the foundation for the inference capability behind Cursor's more than 500 million model calls per day.
The column/row splitting scheme for tensor parallelism was first proposed by Megatron-LM
Sequence parallelism was formally proposed by Megatron-LM in a 2022 paper
In Megatron-LM's tensor parallelism scheme, the Query, Key, and Value projection matrices are split by column, the output projection matrix is split by row, and an All-Reduce communication is inserted at each end
PyTorch's torch.save/torch.load, Hugging Face Transformers' Trainer class, DeepSpeed, and Megatron-LM all have built-in periodic checkpoint support.
Large model training involves multiple stages including data cleaning, annotation, alignment, and safety fine-tuning
Previously, multi-device support relied primarily on upper-level frameworks (such as Megatron-LM) to implement independently; TensorRT has internalized this as a native runtime capability.
For models with parameter counts above 70B, a single machine or single node's VRAM is often unable to accommodate the full model, making Megatron-style model parallelism almost a mandatory choice.
NeMo provides advanced parallel training strategies such as model parallelism and pipeline parallelism by integrating Megatron-LM.
Megatron-LM implements a 3D parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism, while DeepSpeed distributes optimizer states across multiple nodes via ZeRO technology.
All Facts (11)
Megatron-LM is a large model training framework developed by NVIDIA, focusing on tensor parallelism and pipeline parallelism.
70%UnverifiedDuring the CAD phase, the team forked Megatron-LM and Microsoft DeepSpeed and rewrote kernels, laying the foundation for the inference capability behind Cursor's more than 500 million model calls per day.
50%UnverifiedIn Megatron-LM's tensor parallelism scheme, the Query, Key, and Value projection matrices are split by column, the output projection matrix is split by row, and an All-Reduce communication is inserted at each end
50%UnverifiedSequence parallelism was formally proposed by Megatron-LM in a 2022 paper
50%UnverifiedThe column/row splitting scheme for tensor parallelism was first proposed by Megatron-LM
50%UnverifiedPyTorch's torch.save/torch.load, Hugging Face Transformers' Trainer class, DeepSpeed, and Megatron-LM all have built-in periodic checkpoint support.
50%UnverifiedLarge model training involves multiple stages including data cleaning, annotation, alignment, and safety fine-tuning
50%UnverifiedPreviously, multi-device support relied primarily on upper-level frameworks (such as Megatron-LM) to implement independently; TensorRT has internalized this as a native runtime capability.
50%UnverifiedNeMo provides advanced parallel training strategies such as model parallelism and pipeline parallelism by integrating Megatron-LM.
50%UnverifiedFor models with parameter counts above 70B, a single machine or single node's VRAM is often unable to accommodate the full model, making Megatron-style model parallelism almost a mandatory choice.
50%UnverifiedMegatron-LM implements a 3D parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism, while DeepSpeed distributes optimizer states across multiple nodes via ZeRO technology.
50%