UpLoop: Training Recurrent Transformers with Zero Extra Compute, Significant Performance Gains

Recurrent language models enhance reasoning ability by repeatedly computing through shared weights, increasing execution depth without adding independent parameters. But each additional forward pass increases training floating-point operations (FLOPs), making these gains more expensive to obtain. To address this, researchers propose UpLoop—a pretraining scheme that trains recurrent Transformers using upcycling, fully matching a standard dense baseline in final parameter count, total training tokens, and training FLOPs.

The Compute Dilemma of Recurrent Transformers

The core idea of recurrent Transformers is to let the model pass through the same set of shared-weight layers multiple times, simulating the computational effect of a deeper network. This design performs well on reasoning tasks because repeated computation effectively gives the model more "thinking" time. However, the compute overhead during training cannot be ignored: each additional loop means a full forward and backward pass, and FLOPs grow linearly with the number of loops.

This creates a contradiction: stronger reasoning requires more loops, but more loops drive up training costs. Existing approaches tend to trade off between parameter efficiency and training efficiency, making it hard to achieve both. UpLoop's starting point is precisely to break this trade-off—obtaining the performance benefits of recurrent structure without increasing total training FLOPs.

In recurrent Transformers, "recurrence" refers to passing the input sequence through the same set of Transformer layers (typically an intermediate block) multiple times. Each pass is called a "recurrent iteration" or "forward pass." In standard Transformers, each layer has independent parameters; in recurrent Transformers, these layers share weights, so the model's total parameter count does not increase with the number of loops. This design draws on the idea of deep recursive networks, enhancing the model's expressive capacity by increasing computational depth (rather than parameter depth). However, training requires computing gradients for each recurrent iteration, causing the chain rule of backpropagation to unfold along the recurrence, significantly increasing memory and compute overhead during training. UpLoop's core challenge is to let the model learn to exploit recurrent computation without increasing total FLOPs.

How UpLoop Achieves "Zero Extra Compute"

According to the paper, UpLoop employs a progressive training schedule. Early in training, the model is a shallower Transformer with a lighter computational burden. As training progresses, model depth is gradually increased by duplicating already-learned layers. The key is that the compute saved by the early shallow model is "stored" and used in later training to repeatedly execute the shared intermediate block.

In other words, UpLoop redistributes the training process: less compute early, more recurrence later. Ultimately, it fully aligns with the standard dense baseline across three dimensions: parameter count, training tokens, and training FLOPs. This design requires no additional pretraining compute, yet enables the model to possess recurrent execution capability by the end of training.

A progressive training schedule is a strategy that dynamically adjusts model structure or computation graph during training. Common approaches include gradually increasing network depth, width, or input resolution. UpLoop's specific approach is: training begins with a shallower model (e.g., only a fraction of standard depth), where each training step has lower FLOPs; as training proceeds, depth is increased by duplicating trained layers, and recurrent iterations are gradually introduced. This way, FLOPs saved early are accumulated and used for later training of the recurrent block. The final total FLOPs match the standard dense baseline. This schedule requires careful design of duplication timing and recurrence introduction points to avoid training instability.

Experimental Results: Average Downstream Score Improves from 22.99 to 27.52

Experimental results cover multiple model scales. At the 1.3B parameter scale, the average score across 22 downstream tasks improved from the baseline's 22.99 to 27.52—a substantial gain. This improvement is reflected not only in language modeling metrics but also in actual downstream task performance.

More notably, after supervised fine-tuning, the gains from UpLoop extend to mathematical reasoning tasks. This means the computational capability strengthened by the recurrent structure is transferable, not merely effective for optimizing the loss function during pretraining.

Supervised Fine-Tuning (SFT) refers to additional training of a pretrained model on labeled data to adapt it to specific tasks. In UpLoop's experiments, after SFT, the model's performance on mathematical reasoning tasks improved, indicating that the enhanced computational capability from the recurrent structure can transfer to tasks requiring multi-step reasoning. Mathematical reasoning tasks typically require the model to perform chain-of-thought reasoning, and the extra "thinking time" provided by recurrent computation is precisely helpful for such tasks. Note that the choice of SFT data and tasks affects transfer effectiveness, and whether UpLoop's gains broadly apply to other reasoning tasks still requires verification.

Implications for the Training Paradigm of Recurrent Models

UpLoop's value lies in offering a new training approach: the high compute cost of recurrent models is not unavoidable—through redesigning the training schedule, additional benefits can be obtained within an unchanged total budget. This is especially meaningful for research teams with limited resources—they can train models with stronger reasoning capabilities without increasing GPU cluster size.

Of course, the method has so far mainly validated language modeling and mathematical reasoning scenarios, and its generalization to broader tasks remains to be further observed. But at least based on existing data, UpLoop has taken a solid step toward making recurrent Transformers practical.

One-line Summary

UpLoop trains recurrent Transformers via a progressive training schedule with unchanged total FLOPs, enhancing reasoning ability.

Paragraph Summary

UpLoop is a pretraining scheme for training recurrent Transformers, aimed at solving the problem of training compute growing linearly with the number of loops in recurrent models. It employs a progressive depth-growing strategy: early training uses a shallow model to reduce FLOPs, while later stages duplicate layers and introduce recurrent iterations, using the compute saved early for repeated execution of the shared intermediate block. Ultimately, it fully aligns with the standard dense baseline in parameter count, training tokens, and total FLOPs. At the 1.3B parameter scale, the average score across 22 downstream tasks improves from 22.99 to 27.52, and after supervised fine-tuning, mathematical reasoning tasks also benefit. This method provides a training paradigm for resource-limited research teams to obtain stronger reasoning capabilities without increasing compute budgets.