Reinforcement Learning Unlocks Multi-Task Self-Evolution for Generalist Robot Policies

Generalist robot policies allow a single large model to learn multiple tasks from massive pretraining data, with impressive zero-shot generalization capabilities. But when it comes to real-world deployment, these models often still require an adaptation phase to achieve sufficient performance. A recent RSS paper, Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement, points out that iterative self-improvement based on reinforcement learning (RL) holds promise for this adaptation, but previous such methods have mostly been explored only in single-task settings—the goal being merely to improve one narrow deployment task, in stark contrast to the multi-task capabilities of base policies.

The core question of this work: Can generalist robot policies achieve multi-task, generalized improvement during deployment—simultaneously across many different tasks? What algorithmic recipe enables this?

From Single-Task to Multi-Task: Redefining "Improvement"

Previous self-improvement research focused on single tasks, i.e., given one deployment task, continuously optimizing the policy through online interaction. But the value of generalist policies lies precisely in their cross-task capabilities; if the improvement process targets only a single task, it may destroy the base model's original generalization.

The authors argue that true "generalized improvement" should simultaneously enhance performance on multiple tasks and must not come at the expense of tasks not involved in training. This requires algorithm design to account for the diversity of the task distribution rather than optimizing each task in isolation.

To systematically evaluate this capability, the research team first built a structured benchmark to test generalized improvement effects, and conducted a systematic evaluation around algorithmic and practical design choices such as value estimation, policy extraction, and resource allocation.

Catastrophic forgetting is a classic problem in continual learning with neural networks: when a model updates its parameters on new data or new tasks, it often overwrites previously learned knowledge, causing a sharp decline in performance on old tasks. In online RL improvement of robot policies, this risk is particularly prominent—if optimization targets only a single deployment task, the generalization capability acquired by the base model during multi-task pretraining may be destroyed. The "generalized improvement" discussed in this paper is essentially exploring a path to avoid catastrophic forgetting: by having the improvement process cover a diverse task distribution, policy updates do not deviate too far from original capabilities, thereby improving target performance while preserving or even enhancing adaptability to untrained tasks.

Key Recipe: Value-Based RL and Training Task Distribution

Experiments show that generalist policies can not only achieve online improvement on diverse tasks, but the right recipe can also bring generalization—performance improves even on tasks not involved in online training. The authors characterize the conditions under which this generalization occurs and identify two key factors: the training task distribution and the use of value-based RL.

The training task distribution determines the task diversity the policy is exposed to, while value-based RL provides a more stable learning signal to the policy through value function estimation. Combined, these allow the policy during improvement to both specialize in deployment tasks and maintain or even enhance adaptability to unknown tasks.

This finding challenges the intuition that "RL improvement necessarily leads to catastrophic forgetting," offering new ideas for the continual evolution of robot foundation models.

Value-based reinforcement learning (value-based RL) differs from policy gradient methods in that it first learns a value function (such as a Q-function) to estimate the long-term return of states or state-action pairs, and then extracts a policy based on that value function. Common value-based algorithms include DQN and the critic component in SAC (Soft Actor-Critic). In continuous robot control scenarios, the advantage of value-based RL is that the value function provides a dense and relatively stable learning signal, which can alleviate the high variance problem of policy gradient methods and make multi-task online updates more stable. This paper emphasizes value-based RL as one of the key recipes precisely because accurate value estimation helps policies share learning signals across diverse tasks, rather than each chasing high-variance gradients independently.

Structured Benchmark and Design Choices

The authors designed a structured benchmark to quantify generalized improvement capability and systematically compared the impact of different design choices. The evaluation covers value estimation methods, policy extraction approaches, and computational resource allocation strategies.

Results show that subtle differences in the algorithmic recipe can significantly affect generalization performance. For example, the accuracy of value estimation directly relates to the direction of policy updates, while resource allocation determines the efficiency of multi-task learning. These practical insights are instructive for applying RL to real-world robot deployment.

According to the authors, this is one of the first pieces of evidence on online generalized improvement of robot foundation models. The work opens a new direction for post-deployment continual learning of generalist robot policies and suggests that future work should further explore design principles for task distributions and the scalability of RL algorithms.

Policy extraction refers to the process of deriving the actual execution policy from a learned value function or action-value distribution. In value-based RL, common approaches include greedy selection (argmax Q), soft maximization (softmax over Q), or distillation through a separate policy network. The policy extraction method directly affects the selectivity and exploration of behavior: too greedy may cause the policy to become rigid when switching between tasks, while too stochastic may harm learned skills. This paper lists it as one of the design dimensions for systematic evaluation, indicating that in multi-task generalized improvement, how to stably extract policies from value estimates is as important as value estimation itself.

Implications for Robot Foundation Models

This research advances reinforcement learning from single-task fine-tuning to multi-task generalized improvement, meaning that robot foundation models can, after deployment, continuously improve through practice like humans—and the improvement is not limited to the current task. For generalist robots that need to adapt to open environments, this online self-improvement capability is crucial.

Of course, the work is still in its early stages, and there remains a gap between the scale of evaluation and the complexity of real-world scenarios. But the key role it reveals for value-based RL and task distributions provides a reproducible starting point for future research.