DeepSeek-V4 Flash RL Training Successfully Ported to AMD MI355X GPU: A Complete Breakdown

Background: RL Training Moves Toward a Diverse Hardware Ecosystem

For a long time, reinforcement learning (RL) training for large models has been almost entirely dominated by NVIDIA's GPU ecosystem. But with the continued maturation of AMD's Instinct series of accelerators and the ROCm software stack, this landscape is quietly changing. Recently, the Miles team, together with AMD, announced that they have successfully ported the full RL training of DeepSeek-V4 Flash onto AMD Instinct MI355X GPUs, achieving stable end-to-end operation.

ROCm (Radeon Open Compute) is AMD's open-source GPU computing platform. Through the HIP (Heterogeneous-compute Interface for Portability) programming model, it offers developers a CUDA-like experience, allowing CUDA code to be ported to AMD hardware with relatively low friction. It's worth noting that ever since ROCm's first release in 2016, it has continuously faced a substantial gap with the CUDA ecosystem—CUDA has accumulated nearly 20 years of development, boasting an enormous set of optimized libraries (cuDNN, cuBLAS, NCCL, etc.), a mature debugging toolchain, and a developer community numbering in the millions. The CUDA compatibility ROCm achieves through its HIP layer has long faced practical challenges, including operator performance discrepancies, insufficient communication library stability, and lagging third-party library support. ROCm 6.x, released in 2024, introduced hardware-level support for FP8, improved RCCL (ROCm Collective Communications Library) communication performance, and more refined MoE model operator optimizations. These improvements form the key software foundation that made this porting effort successful, and they represent significant progress for ROCm in terms of ecosystem completeness and stability.

The significance of this milestone lies not merely in "getting it running," but in the fact that the team, through systematic engineering validation, has proven the feasibility and stability of large-scale RL training on the ROCm platform. For enterprises and research institutions hoping to break free from dependence on a single hardware vendor and build a diversified compute supply chain, this is an important signal worth watching.

Core Results: From Alignment Validation to Model Convergence

The central challenge of this porting effort was ensuring behavioral consistency of the model across two engines: SGLang rollout (inference sampling) and Megatron training (distributed training). The team completed validation across more than 100 optimizer iteration steps on four MI355X nodes, achieving several key results.

Numerical Alignment Precision Between Training and Sampling

In RL training, mainstream algorithms such as PPO (Proximal Policy Optimization) rely on the log-probabilities collected during the rollout phase to compute the importance sampling ratio. Understanding the mathematical essence of this mechanism helps illustrate why alignment work is so important: at its core, PPO reuses rollout data through importance sampling, with its ratio r(θ) = π_θ(a|s) / π_θ_old(a|s)—the probability ratio between the current policy and the sampling policy. When the rollout engine and the training engine compute systematically divergent log-probs for the same token sequence, this ratio becomes distorted. Excessive divergence causes PPO's clip mechanism to fail or introduces spurious policy-update signals, and in severe cases can cause training to diverge.

Through meticulous engineering alignment, the Miles team kept the train-rollout log-prob gap stably within a range of approximately 0.09 across more than 100 steps. Mathematically, an average divergence of 0.09 means the numerical computation error between the two engines does not exceed roughly 9%, which falls within the acceptable statistical-noise range for sampling-based policy gradient estimation and does not substantively distort gradient estimates—laying a solid foundation for stable subsequent training.

Quantifiable Improvement in Model Capability

Even more noteworthy, the training yielded quantifiable gains in reasoning capability. On the mathematical reasoning benchmark AIME-2024:

• pass@1 improved from 0.39 to 0.49
• pass@8 improved from 0.53 to 0.67

This means that RL training for DeepSeek-V4 Flash did not just "run" on AMD hardware—it genuinely delivered effective improvements in the model's reasoning ability, validating the practical effectiveness of the entire training pipeline.

Key Techniques: Mixed Precision and Distributed Parallelism Strategies

Datatype-Aware Online Weight Updates

This work employed a mixed-precision scheme of FP8 rollout paired with a BF16 actor. FP8 is an 8-bit floating-point format for which NVIDIA first introduced hardware support in the H100 architecture. It comes in two subtypes: E4M3 (4-bit exponent + 3-bit mantissa, suited for forward propagation) and E5M2 (5-bit exponent + 2-bit mantissa, suited for gradient computation). The AMD MI355X likewise natively supports FP8 computation at the hardware level—an important capability upgrade over the previous-generation MI300X. Compared to BF16 (Brain Float 16), FP8 can process roughly twice the data volume under the same memory bandwidth, and matrix-multiplication throughput can theoretically double, so FP8 is used in the inference sampling phase to achieve higher throughput efficiency.

However, FP8 has an extremely narrow dynamic range (E4M3 can only represent values of roughly ±448) and requires careful scaling-factor management and overflow detection mechanisms. It is prone to numerical overflow or underflow during backpropagation and gradient accumulation, so the actor model on the training side maintains BF16 precision—BF16 balances sufficient dynamic range with computational efficiency and is the mainstream precision choice for current large-model training. To coordinate weight synchronization between the two different precisions, the team implemented a "datatype-aware online weight updates" mechanism, striking an optimal balance between computational efficiency and numerical precision.

Stable Parallel Layout on the ROCm Platform

At the distributed training level, the team successfully ran a TP1 / PP4 / EP4 parallel layout on the ROCm platform. Each of these three dimensions has its own role, and they also reflect the engineering trade-offs made on the ROCm platform:

Tensor Parallelism (TP) splits the computation matrix of a single operator across multiple GPUs. It involves frequent, latency-sensitive communication and is usually confined to GPUs interconnected via NVLink within the same node; TP1 means tensor parallelism is not enabled—this choice was likely made precisely to reduce the debugging complexity of high-frequency communication links on ROCm, prioritizing stability. Pipeline Parallelism (PP) distributes different layers of the model across different nodes, with low communication volume but the overhead of pipeline bubbles; PP4 means the model is split into 4 pipeline stages. Expert Parallelism (EP) is a parallelism dimension unique to MoE models, requiring All-to-All communication to route tokens; EP4 means the experts are distributed across 4 parallel units. The TP1/PP4/EP4 combination reflects an engineering trade-off that prioritizes communication stability on the ROCm platform.

It's especially worth emphasizing that no collective stalls occurred throughout the entire run. Stalls caused by cross-node synchronization operations such as AllReduce and AllGather have historically been a chronic ailment of non-NVIDIA platform training. This result demonstrates that ROCm's communication library (RCCL) is now production-ready in real large-scale training scenarios.

Architecture Alignment: Ensuring Consistency of Complex Model Components

DeepSeek-V4 Flash incorporates several cutting-edge architectural designs, including Hybrid Attention, the mHC hybrid mechanism, and hash-routed MoE. Hybrid Attention combines standard multi-head attention with other variants (such as linear attention or sliding-window attention) within the same model, balancing computational complexity and expressive power for long-sequence processing.

Hash-routed MoE is a particularly noteworthy design: traditional MoE uses a learnable gating network to decide token routing, which is prone to Router Collapse—a phenomenon where a large number of tokens are routed to the same small set of experts while others sit idle. Hash routing performs a hash operation on the token's positional or content features to deterministically assign each token to a fixed subset of experts, completely bypassing learnable gating and fundamentally eliminating router collapse. However, this deterministic design also imposes a strict constraint: the hash function implementations in the inference engine and the training engine must be bit-for-bit identical. Any implementation discrepancy would activate different expert weights, producing entirely unpredictable numerical differences and greatly increasing the difficulty of log-prob alignment.

The implementations of these complex components in the inference engine and the training engine must be strictly aligned; any minor behavioral discrepancy could be progressively amplified throughout the long chain of RL training. The team confirmed that all the aforementioned key components achieved behavioral alignment between the SGLang and Megatron engines. This is also the most engineering-intensive part of the entire project—it requires a deep understanding of the numerical behavior of each operator across different frameworks and different hardware backends, along with fine-grained tuning.

Industry Significance: A Landmark in the Diversification of the Compute Ecosystem

From a broader perspective, the successful deployment of DeepSeek-V4 Flash on the AMD MI355X is a landmark event in the diversification of the large-model training compute ecosystem.

On one hand, it reduces dependence on a single hardware vendor, providing more strategic options for compute procurement. On the other hand, it further validates the maturity of the ROCm software stack for the most cutting-edge RL training tasks. For the AI infrastructure field as a whole, healthy competition at the hardware level will ultimately translate into lower training costs and faster iteration speeds.

Of course, there is still a considerable distance between "validating 100+ steps" and "supporting full production-grade RL training." Stability, convergence behavior over long-running sessions, and scalability across larger clusters all remain directions that require ongoing validation. Regardless, this step has already opened up new possibilities, and the AMD MI355X and the ROCm ecosystem have now officially entered the competitive arena of large-model reinforcement learning training.