Megatron Core
An open-source large-scale model training framework from NVIDIA that supports three-dimensional parallelism strategies, including tensor parallelism, pipeline parallelism, and data parallelism. It features bit-level deterministic training and is widely used for pretraining very large-scale language models.
Timeline (last 90 days)
NVIDIA introduced bitwise determinism training capability through Megatron Core.
Megatron Core's design philosophy is to make determinism a configurable option, enabled during debugging and validation phases and turned off as needed during production training that pursues maximum throughput.
Megatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.
Achieving bit-level determinism requires the framework to strictly fix the execution order of aggregation operations, at the cost of sacrificing some overlap optimization space between communication and computation.
Megatron Core is a widely used open-source training framework.
All Facts (5)
Megatron Core's design philosophy is to make determinism a configurable option, enabled during debugging and validation phases and turned off as needed during production training that pursues maximum throughput.
50%UnverifiedMegatron Core adopts a three-dimensional parallelism strategy combining tensor parallelism, pipeline parallelism, and data parallelism.
50%UnverifiedAchieving bit-level determinism requires the framework to strictly fix the execution order of aggregation operations, at the cost of sacrificing some overlap optimization space between communication and computation.
50%UnverifiedMegatron Core is a widely used open-source training framework.
50%UnverifiedNVIDIA introduced bitwise determinism training capability through Megatron Core.
50%