LM2LM: Language Models Learn Self-Improvement, Opening a New Path to Recursive Self-Improvement

Title

LM2LM: Language Models Learn Self-Improvement, Opening a New Path to Recursive Self-Improvement

Meta Description

LM2LM proposes using reinforcement learning to train language models to generate model improvement proposals, using a world model as a reward model and calibrating feedback through periodic real execution. Across eight development environments, Qwen3.8-27B surpasses GPT-6-sol with 33% lower data generation cost.

Article Content

Large language models (LLMs) are increasingly being used to automate AI research, including the development of language models themselves. This points to an enticing path: recursive self-improvement—models capable of proposing improvements that make the next generation of models stronger. However, translating experimental feedback into a learnable, cross-model-effective capability for generating improvement proposals still faces two major challenges. A recent paper's LM2LM method trains LLMs via reinforcement learning (RL) to generate effective model improvement proposals, using a "research world model" as the reward model, and achieves notable results across multiple development environments.

Challenge: Real Feedback Is Expensive, Proxy Feedback Is Unreliable

Training model improvement capability directly with real experimental feedback is extremely costly. RL requires evaluating tens of thousands of proposals, and each proposal may require fully training a model once. Even if each training run takes only 10 GPU-hours, collecting rewards would consume hundreds of thousands of GPU-hours. This motivates researchers to seek cheaper proxy signals.

But proxy feedback has reliability problems: as the policy is optimized, the proposal distribution shifts, and the policy tends to favor proposals that are overestimated, causing the proxy reward to become distorted. How to obtain scalable feedback without sacrificing reliability is the core tension that LM2LM aims to resolve.

Reward Models and Reward Hacking in Reinforcement Learning

In reinforcement learning (RL), when real environment rewards are difficult to obtain frequently, a learnable proxy model is often used to approximate the reward function; such models are collectively called reward models. The goal of RL training is to maximize cumulative reward, so the policy actively seeks outputs that the reward model scores highly. The problem is that there is always a gap between the proxy reward and the true reward, and the more thoroughly the policy is optimized, the easier it is to exploit this gap—generating samples that score highly according to the proxy model but whose true quality has not improved or has even declined. This phenomenon is called reward hacking; in RLHF, it manifests as model outputs catering to the preference model rather than true human preferences. The "unreliable proxy feedback" faced by LM2LM is a concrete instance of reward hacking: the policy concentrates on proposals that the research world model (RWM) overestimates, gradually distorting the reward signal.

LM2LM: Using a World Model as Judge, with Periodic Calibration

The core design of LM2LM consists of two parts. First, it uses a frozen LLM as the "research world model" (RWM), which predicts the payoff of proposals based on real experimental records provided in context, and uses this as the RL reward model. In this way, obtaining feedback no longer relies on large amounts of real training, but instead performs inference based on existing experimental records.

Second, to maintain feedback reliability, LM2LM periodically executes the top-ranked proposals under the current policy that are overestimated by the RWM, and adds the actual measurement results to the RWM's context. This is equivalent to continuously calibrating the world model with real execution results, preventing the policy from over-optimizing the proxy reward.

According to the paper, in eight autoregressive and diffusion language model development environments, real execution evaluations show that LM2LM enables Qwen3.8-27B to surpass GPT-6-sol—even though the latter received 300 real experimental records in context. At the same time, LM2LM reduces the required data generation cost by 33%. In addition, LM2LM can generalize to model families unseen during training.

A Frozen LLM as a World Model with In-Context Learning

The "research world model" (RWM) referred to by LM2LM is not a dedicated predictive model trained on environment dynamics, but a parameter-frozen LLM that performs prediction through in-context learning: several real "proposal—execution result" records are placed in the prompt as examples, and the model then predicts the payoff of new improvement proposals. The advantage of this design is that there is no need to separately collect training data or fine-tune parameters for the reward model; it directly reuses the LLM's reasoning and pattern-recognition capabilities. The cost is that prediction quality depends heavily on the representativeness and number of in-context examples. When new proposals generated by the policy deviate from the distribution of existing examples in context, prediction error rises significantly—which is exactly why real execution results need to be periodically injected to "refresh" the context.

Significance and Limitations from Multiple Perspectives

Results from a single source are impressive, but we need to view them rationally. LM2LM demonstrates a possibility: language models can not only write code and tune parameters, but also learn how to propose high-value model improvement plans, and this capability can transfer across models. This provides a new idea for automated AI research—using a world model to reduce trial-and-error costs, and using periodic real execution to ensure the direction is correct.

However, the method has so far been validated only in eight development environments and on a limited set of models; the complexity and diversity of real-world scenarios are far greater. In addition, how the world model's prediction accuracy and calibration frequency affect final performance is not elaborated in detail in the paper. In the future, how to extend LM2LM to larger-scale and more modality-diverse model development remains an open question.

Overall, LM2LM takes a solid step toward recursive self-improvement. It does not aim for full model autonomy, but instead designs a human-machine collaborative feedback loop: the model proposes proposals, the world model screens them, and real execution calibrates them. This pragmatic design may be closer to deployment than aggressive "fully automated" approaches.

Recursive Self-Improvement and Automated AI Research

Recursive self-improvement refers to a system's ability to improve its own improvement capability, thereby forming a feedback loop of capability growth; it is one of the core concepts in AI safety discussions. In the LLM context, it need not mean the model directly modifying its own weights; it can manifest as the model proposing improvements at the level of architecture, data, or training recipes, which are then validated and executed by external processes. The more realistic form of current research is "AI-assisted AI research": models handle proposal generation, experiment screening, and other steps, while real training and evaluation are still completed by engineering pipelines. The key bottleneck of this path is not whether proposals are novel, but the cost and reliability of validating them—which also explains why LM2LM focuses on replacing expensive real experiments with low-cost proxy rewards and designing calibration mechanisms to prevent proxy signal drift.

One-line Summary

LM2LM uses reinforcement learning to train LLMs to propose model improvements, with a world model as reward and periodic real calibration.

Paragraph Summary

LM2LM aims to train large language models to generate effective model improvement proposals, thereby advancing recursive self-improvement. Directly using real experimental feedback for reinforcement learning is too costly, since evaluating each proposal may require fully training a model once; cheap proxy feedback, however, becomes distorted due to policy distribution shift, causing reward hacking. LM2LM's solution is to use a frozen LLM as a "research world model" that predicts proposal payoff based on real experimental records in context and serves as the reward model, while periodically executing top-ranked and overestimated proposals and feeding measured results back into the context to calibrate the world model. In eight autoregressive and diffusion language model development environments, the method enables Qwen3.8-27B to surpass GPT-6-sol with 300 real experimental records, reduces data generation cost by 33%, and generalizes to unseen model families. The authors consider this a pragmatic step toward recursive self-improvement, but the experimental scale is limited, and the impact of world model accuracy and calibration frequency remains to be studied.