FP8 Training Lands on AMD GPUs: TorchTitan Performance Optimizations Merged Upstream

AMD GPUs Enter the FP8 Training Mainstream

NVIDIA's CUDA ecosystem has long dominated the deep learning training landscape. While AMD Instinct GPUs have continued to push hardware performance boundaries, they've consistently faced challenges in software ecosystem compatibility. At PyTorch Conference 2025, AMD announced a development that marks a significant step toward changing this dynamic.

According to AMD's official blog, the team demonstrated linear scaling across more than 1,000 GPUs on AMD Instinct clusters — powered by Primus-Turbo, AMD's optimization library built for training frameworks like TorchTitan. More importantly, AMD has formally merged these optimizations upstream into the official PyTorch repositories, enabling TorchTitan to natively support AMD Instinct GPUs with competitive FP8 training performance right out of the box.

Why FP8 Training Matters

Balancing Precision and Efficiency

FP8 (8-bit floating point) has emerged as a widely discussed low-precision numeric format for large model training. Compared to traditional FP16 or BF16, FP8 can significantly reduce memory usage, boost compute throughput, and lower communication overhead — all while maintaining acceptable training accuracy. For large language models with hundreds of billions of parameters, this translates directly into lower costs and faster iteration cycles.

However, FP8 training places demanding requirements on both hardware and the software stack. Until now, mature FP8 training support was largely confined to NVIDIA's Hopper architecture GPUs, leaving AMD users unable to enjoy the same efficiency gains. AMD's integration of FP8 optimizations into TorchAO and TorchTitan directly addresses this gap.

The Significance of 1,000+ GPU Linear Scaling

Linear scaling is a critical benchmark for distributed training systems. Ideally, doubling the number of GPUs should nearly double training throughput. In practice, as cluster size grows, communication bottlenecks and synchronization overhead often cause scaling efficiency to degrade. AMD's demonstration of linear scaling at over 1,000 GPUs validates the maturity of its hardware interconnect capabilities and software optimizations, offering a reliable alternative for large-scale training workloads.

The Strategic Value of Upstream Merging

From Proprietary Optimizations to Open Standards

AMD has emphasized that all relevant contributions have been merged into the upstream pytorch/AO and pytorch/TorchTitan repositories. This is particularly significant. Historically, hardware vendor optimizations tended to live as independent forks or proprietary libraries — requiring extra configuration, carrying high maintenance costs, and making it difficult to benefit from mainline updates.

Merging optimizations directly upstream into official PyTorch projects means:

• Out-of-the-box usability: Users no longer need to hunt for third-party patches — installing standard PyTorch is enough to access AMD FP8 training capabilities
• Ongoing maintenance: Code that enters the mainline will be continuously updated and maintained with each PyTorch release
• Ecosystem recognition: This also reflects the PyTorch community's growing commitment to AMD hardware support

The Roles of TorchTitan and TorchAO

TorchTitan is PyTorch's official large-scale training reference implementation, focused on demonstrating how to efficiently train large models using native PyTorch features. TorchAO is PyTorch's architecture optimization library, handling performance features like quantization and low-precision computation. By contributing to both of these core projects, AMD has established a meaningful presence at critical nodes within the PyTorch training ecosystem.

Potential Impact on the Industry Landscape

Breaking Single-Vendor Dependency

For years, AI training compute has been heavily concentrated in the hands of a single vendor — creating procurement cost pressures and supply chain vulnerabilities. If AMD Instinct GPUs can deliver performance and ease of use comparable to competing products on mainstream training frameworks, enterprises and research institutions will have a genuinely viable path to diversifying their compute resources.

Software Ecosystem Is the Real Moat

This development reinforces a widely held industry belief: in the AI hardware race, peak compute performance matters, but what ultimately drives adoption is the completeness of the software ecosystem. AMD's approach — entering through the Primus-Turbo optimization library and then integrating into the PyTorch mainline via upstream merges — represents a pragmatic, ecosystem-first strategy.

Closing Thoughts

AMD's upstream contribution of FP8 training optimizations to TorchTitan and TorchAO represents an important move in its broader effort to build a foothold in the AI training ecosystem. The demonstrated linear scaling across 1,000+ GPUs, combined with out-of-the-box FP8 support, meaningfully strengthens AMD Instinct GPUs' competitiveness for large model training. For teams seeking to diversify their compute options, this is a signal worth paying attention to. As more optimizations continue to land in the PyTorch mainline, AMD's position in the training ecosystem may continue to solidify.