A Single Algorithm Unifies the Entire LLM Training Pipeline

Large model training is typically split into four stages: pre-training, mid-training, supervised fine-tuning (SFT), and reinforcement learning (RL). Each stage not only uses a different data distribution but also a different learning algorithm. This staged paradigm has become almost default engineering common sense in the industry. But a new study poses a fundamental question: is separation at the algorithmic level actually necessary?

Where Does the Inertia of Staged Training Come From?

The reason today's foundation model training pipelines are staged is largely a product of history and engineering compromise. The pre-training stage uses massive static corpora and performs next-token prediction with maximum likelihood as the objective; the SFT stage switches to human-annotated instruction data, where the objective form looks the same but the nature of the data has changed; the RL stage introduces reward models and policy gradients, and the algorithmic framework switches entirely.

Each segment has its own hyperparameter tuning, its own monitoring metrics, and its own infrastructure requirements. The benefit of staging is that each segment can be optimized independently; the drawback is that the transitions between stages often rely on empirical rules, and data mixing ratios and switch timing lack unified, principled guidance. The researchers point out that this algorithmic separation is not fundamental but rather an inherited habit.

According to the study, treating different stages as a combination of different algorithms makes the entire training lifecycle difficult to analyze within a coherent theoretical framework. What data should enter training at what stage and in what proportion becomes a pile of hyperparameters requiring experimental trial and error.

Unified Training: Writing the RL Objective as Maximum Likelihood

The core idea of this work is to derive the RL objective as a maximum likelihood fit to the model's generated target distribution. Once the RL stage's objective can also be written in likelihood form, it falls mathematically into the same family of objective functions as pre-training and SFT. Based on this, the authors propose a unified training framework that incorporates all the above stages into a single likelihood-based training algorithm.

The direct corollary of this perspective is that the entire training lifecycle is no longer a question of "switching algorithms" but a question of "choosing what data to train on, in what proportion, and at what timing." Traditional staged training degenerates into a special case under the unified framework—merely a particular arrangement of data selection and mixing strategy.

In other words, the boundaries between pre-training, mid-training, SFT, and RL shift from hard switches at the algorithmic level to continuous decisions at the data scheduling level. This provides a more unified language for designing training strategies.

Maximum likelihood estimation (MLE) is one of the most fundamental parameter estimation methods in statistics: given a set of observed data, choose the model parameters that make those data most probable. In language model pre-training, it is concretized as minimizing the cross-entropy loss, i.e., maximizing the model's predicted probability of the true next token. The SFT stage is similar in form, except the data is replaced with instruction-response pairs.

Writing the RL objective in maximum likelihood form is mathematically non-trivial. Standard RL (such as PPO) optimizes expected reward, and gradient estimation relies on the policy gradient theorem, typically requiring techniques such as importance sampling and baseline subtraction, which differ greatly from the gradient form of supervised learning. Recent research (such as direct preference optimization, DPO) has shown that under specific reward structures, the RL objective can be equivalently transformed into likelihood maximization over some target distribution, thereby bypassing explicit reward models and policy gradients. The idea in this paper is precisely to generalize this reduction to the entire training lifecycle, so that RL shares the same optimization language as pre-training and SFT.

System Implementation: Static Data and RL-Generated Data Mixed in the Same Trainer

A mathematical framework alone is not enough; the authors further build a unified training system that can mix static data and RL-generated data within a single trainer. This is quite critical engineering-wise: previously the RL stage required independent sampling and training loops, a separate set of infrastructure from the pre-training data pipeline; the unified training system merges them into the same process.

The paper claims that in controlled experiments and real-world scenario experiments, unified training improves the accuracy-compute tradeoff, expands task coverage, while preserving the model's broad capabilities and providing a stronger starting point for continual learning. These conclusions come from the authors' own experimental evaluation; specific quantitative metrics should be referenced in the original paper.

A detail about the two lines of experimental design—"controlled experiments" and "real-world scenario experiments": the former is used to verify the framework's behavior under ideal conditions, while the latter tests its usability at actual training scale; the two complement each other.

Continual learning refers to a model's ability to continue learning new tasks or new data without forgetting old knowledge after completing initial training. Its core challenge is catastrophic forgetting: parameter updates overwrite previously learned representations. In multi-stage training paradigms, the SFT and RL stages often weaken the general capabilities acquired during pre-training, and subsequent learning of new tasks may cause further degradation. The unified training framework is believed to provide a stronger starting point for continual learning because all stages share the same objective function, and model parameters always remain in the same optimization landscape, so incremental updates do not need to cross objective discontinuities caused by different algorithms. However, this remains a hypothetical advantage that requires experimental evidence.

What It Means for Training Practice

If the conclusions of unified training hold at larger scales, its impact on training practice may manifest at several levels.

At the infrastructure level, the training system needs to support mixing static corpora and online-generated data in proportion within the same loop, and the design focus of data pipelines will shift from "staged switching" to "dynamic mixing scheduling."

At the research level, the paper positions itself as foundational work toward a "unified science of foundation model training," attempting to connect mathematical principles, training infrastructure, and capability development.

At the continual learning level, the authors specifically mention that unified training provides a stronger starting point for continual learning. This echoes the dilemma under the staged paradigm: models after multi-stage training are prone to capability degradation when continuing to learn new tasks, whereas a model trained under a unified objective may be structurally better suited for incremental updates.

It should be noted that these impacts are currently still at the stage of framework proposal and preliminary validation, and there is still a distance to large-scale industrial validation.

Questions Still to Be Answered

Unified training proposes an elegant mathematical reduction, but from reduction to practice there remain several open questions. How should the optimal mixing ratio of static data and RL-generated data be determined? The paper transforms this into a data selection problem, but the selection strategy itself still needs methodological support. How the engineering experience accumulated in staged training—such as staged learning rate scheduling—maps under the unified framework also needs to be re-examined.

In addition, the "preserving broader capabilities" and "stronger continual learning starting point" mentioned in the abstract are qualitative statements; the actual magnitude of gains and whether they are stable across different model scales all require more detailed evidence from future work.

The value of this work lies in changing the way the question is asked: not asking what algorithm each stage should use, but asking whether all stages can share a single algorithm. The answer is yes, and the remaining question becomes the new discipline of data scheduling.

One-line Summary

A new study writes the RL objective as maximum likelihood, using a single algorithm to unify four-stage training: pre-training, SFT, and RL.

Paragraph Summary

The article introduces a study that challenges the staged training paradigm of large models. Currently, the four stages of pre-training, mid-training, SFT, and RL each use different algorithms, and this separation is more a product of history and engineering compromise. The study derives the RL objective as a maximum likelihood fit to the model's generated target distribution, thereby grouping the four stages into the same family of likelihood objectives and proposing a unified training framework: the training problem shifts from "switching algorithms" to "training on what data, in what proportion, and at what timing," and staged training degenerates into a special case of data mixing strategy. The authors also build a system that can mix static data and RL-generated data within a single trainer, with experiments claiming improved accuracy-compute tradeoff, expanded task coverage, and a stronger continual learning starting point. The article also points out open questions such as determining the optimal mixing ratio and mapping engineering experience, which remain to be answered.