Full-Parameter RL on 310B Model Across 1000+ TPUs: The Engineering Breakthrough Behind Xiaomi MiMo-V2.6

Full-Parameter Reinforcement Learning on a Thousand-Chip TPU Cluster

A recent post on Twitter sparked significant industry attention around the engineering capabilities required for large-scale reinforcement learning. The Peano AI team announced they had successfully run full-parameter reinforcement learning (full-parameter RL) on Xiaomi's (@XiaomiMiMo) 310B-parameter MiMo-V2.6 model across a cluster of more than 1,000 TPUs.

What makes this noteworthy is that running full-parameter RL at this scale — rather than fine-tuning a subset of parameters or using lightweight approaches like LoRA — places extreme demands on underlying infrastructure, communication bandwidth, and training stability. For a model at the 310B (310-billion parameter) scale, successfully running the complete RL pipeline on a thousand-chip TPU cluster is a significant systems engineering achievement in its own right.

vLLM-Driven Rollout with Bitwise-Matched Alignment to the Trainer

Based on the details disclosed in the original post, the technical highlights of this system are concentrated in several key areas.

First is the choice of inference engine: vLLM drives the rollout (sample generation) process. During RL training, the rollout phase requires the model to generate large numbers of samples under the current policy — and the throughput efficiency of this step directly determines the speed of the entire training loop. vLLM, as a high-performance inference framework, takes on the critical role of handling generation.

Even more impressive is what the team achieved at verification: bitwise-matched alignment between the rollout and the trainer. This means the model computations used to generate samples during inference are numerically identical to those used to update parameters during training — with no discrepancies introduced by differences in framework implementations or precision handling.

In practice, the trainer and inference engine are typically two separate implementations, and numerical mismatches between them (train-inference mismatch) are a well-known source of training instability and policy degradation. Achieving bitwise-level alignment reflects serious engineering investment in numerical consistency, with direct benefits for training stability and result reproducibility.

Rollout in RL is the core step of the reinforcement learning training loop: the policy model, at its current parameter state, autoregressively decodes a set of input prompts to generate complete output sequences, which are then scored by a reward model, with those scores driving parameter updates via algorithms like PPO or GRPO. For very large models, the rollout phase is often the throughput bottleneck of the entire training loop, since it requires repeated full-model forward passes. vLLM addresses this through mechanisms like PagedAttention and continuous batching, dramatically improving inference throughput and making it feasible to generate sufficient training samples at thousand-chip scale.

The challenge of bitwise matching lies in the fact that training frameworks (e.g., JAX/XLA on TPU) and inference engines (e.g., vLLM) often differ subtly in operator implementations, mixed-precision strategies, and numerical accumulation ordering — causing the same set of weights to produce different output logits on each side. In RL training, this numerical drift directly affects the computation of importance sampling ratios, leading to biased policy gradient estimates, training instability, and even policy collapse. Achieving bitwise consistency typically requires either unifying the underlying operator paths or introducing rigorous numerical validation procedures.

310B Parameters Synchronized Across ICI Interconnect in Under 2 Seconds

Another striking figure is the communication performance: the full 310B parameters are transferred across the ICI (Inter-Chip Interconnect) network in under 2 seconds.

ICI is the high-speed interconnect fabric linking chips within a Google TPU cluster — one of the key architectural differentiators between TPUs and GPU clusters. In the full-parameter RL training loop, parameters must be synchronized between the trainer and the inference engine: after each policy update, the new weights need to be distributed to the instances responsible for rollout. At the 310B scale, if this parameter transfer is not handled carefully, it becomes a serious performance bottleneck.

Compressing the full synchronization of 310 billion parameters into under 2 seconds fully leverages the high-bandwidth characteristics of the TPU ICI fabric, indicating that this pipeline has been deeply optimized at the parameter transfer stage. This also indirectly validates the competitive advantage of TPU interconnect architecture in ultra-large-scale model training scenarios.

ICI (Inter-Chip Interconnect) is Google's high-speed direct chip-to-chip interconnect inside a TPU pod — bypassing host CPUs and traditional Ethernet, with bandwidth on the order of hundreds of GB/s. Unlike the two-tier network topology common in GPU clusters (NVLink within a node, InfiniBand/RoCE across nodes), TPU's ICI forms a unified high-dimensional Torus topology within the pod (such as a 3D or 4D Torus), enabling large-scale collective communication operations like AllReduce and parameter broadcasts to be executed efficiently at the hardware level — with better latency and bandwidth utilization than software-level cross-node communication schemes.

At float16 or bfloat16 precision, 310B parameters occupy roughly 620 GB. Completing a full broadcast synchronization of all parameters in 2 seconds implies an effective bandwidth of over 310 GB/s (for one-directional transfer alone) — a clear demonstration of the ICI fabric's efficiency for the "large message, low-frequency" communication pattern characteristic of parameter transfers. By comparison, cross-node parameter synchronization in GPU clusters is typically constrained by InfiniBand bandwidth ceilings, often requiring significantly more time or more complex pipeline-overlap strategies at the same scale.

Why This Matters

Putting these technical points together, a relatively complete picture of large-scale RL engineering practice emerges: vLLM handles efficient sampling, the trainer handles parameter updates, the two are synchronized rapidly over the ICI high-speed network, and numerical bitwise consistency is maintained throughout.

For the industry, this announcement carries several signals:

• TPUs are becoming a viable option for ultra-large-scale RL training. Discussions of large model training and RL have historically centered on the GPU ecosystem, but successfully running full-parameter RL on a 310B model across 1,000+ TPUs demonstrates an alternative, deployable technical path.
• The engineering depth of domestic Chinese large models is growing. Xiaomi's MiMo series, as a participant in this effort, being able to support RL experiments at this scale reflects the accumulation of large model training infrastructure within domestic teams.
• Train-inference consistency is becoming a central issue in RL engineering. The explicit mention of bitwise matching signals that the industry is increasingly taking seriously a stability risk that has long been underappreciated.

It's worth noting that this announcement represents an early-stage technical disclosure. Specific training outcomes, improvements in model performance, and cost data have not yet been made public. How much capability improvement this work ultimately yields remains to be verified by a more complete technical report or benchmark results.

Summary

Completing full-parameter reinforcement learning on the 310B MiMo-V2.6 model across 1,000+ TPUs is a compelling demonstration of large-scale model systems engineering. The three technical highlights — vLLM-driven rollout, bitwise-aligned train-inference consistency, and sub-2-second ICI synchronization of 310B parameters — together define what makes this pipeline noteworthy. It showcases both the potential of TPU interconnect architecture at extreme scale and reflects a broader trend in RL engineering toward larger models and higher numerical consistency.