UpLoop
A pretraining scheme for training looped Transformers via upcycling, which through a progressive training schedule obtains the performance benefits of the looped structure while keeping total training FLOPs unchanged.
Timeline (last 90 days)
UpLoop is a pretraining scheme that trains a looped Transformer using upcycling, matching a standard dense baseline exactly in final parameter count, total training tokens and training FLOPs.
At the 1.3B parameter scale, UpLoop raises the average score on 22 downstream tasks from the baseline's 22.99 to 27.52.
After supervised fine-tuning (SFT), the gains brought by UpLoop can also extend to mathematical reasoning tasks.
UpLoop's progressive schedule requires carefully designed copy timing and loop introduction points to avoid training instability.
UpLoop has so far mainly validated language modeling and mathematical reasoning scenarios, and its generalization ability on broader tasks remains to be further observed.
All Facts (5)
UpLoop is a pretraining scheme that trains a looped Transformer using upcycling, matching a standard dense baseline exactly in final parameter count, total training tokens and training FLOPs.
50%UnverifiedAt the 1.3B parameter scale, UpLoop raises the average score on 22 downstream tasks from the baseline's 22.99 to 27.52.
50%UnverifiedAfter supervised fine-tuning (SFT), the gains brought by UpLoop can also extend to mathematical reasoning tasks.
50%UnverifiedUpLoop's progressive schedule requires carefully designed copy timing and loop introduction points to avoid training instability.
50%UnverifiedUpLoop has so far mainly validated language modeling and mathematical reasoning scenarios, and its generalization ability on broader tasks remains to be further observed.
50%