Unified Training
A training paradigm that unifies large-model training stages, including pretraining, mid-training, supervised fine-tuning, and reinforcement learning, into a single likelihood-based algorithmic framework. Its core is converting the RL objective into maximum-likelihood fitting so that all stages share one optimization language, shifting training strategy from switching algorithms to decisions about data scheduling.
Timeline (last 90 days)
A new study proposes deriving the RL objective as maximum likelihood fitting to a model-generated target distribution, thereby unifying pretraining, SFT and RL into the same family of objective functions.
The study proposes a unified training framework that incorporates all training stages into a single likelihood-based training algorithm.
The unified training framework is believed to provide a stronger starting point for continual learning because all stages share the same objective function, so model parameters always remain in the same optimization landscape.
The paper turns the optimal mixing ratio of static data and RL-generated data into a data selection problem, but the selection strategy itself still requires methodological support.
The phrases "preserving broader capabilities" and "a stronger starting point for continual learning" mentioned in the abstract are qualitative statements; the actual magnitude of gains and their stability across different model scales require more detailed evidence from subsequent work.
All Facts (5)
A new study proposes deriving the RL objective as maximum likelihood fitting to a model-generated target distribution, thereby unifying pretraining, SFT and RL into the same family of objective functions.
50%UnverifiedThe study proposes a unified training framework that incorporates all training stages into a single likelihood-based training algorithm.
50%UnverifiedThe unified training framework is believed to provide a stronger starting point for continual learning because all stages share the same objective function, so model parameters always remain in the same optimization landscape.
50%UnverifiedThe paper turns the optimal mixing ratio of static data and RL-generated data into a data selection problem, but the selection strategy itself still requires methodological support.
50%UnverifiedThe phrases "preserving broader capabilities" and "a stronger starting point for continual learning" mentioned in the abstract are qualitative statements; the actual magnitude of gains and their stability across different model scales require more detailed evidence from subsequent work.
50%