How to Safely Upgrade Robot Controllers? A New Method Ensures Old Skills Don't Degrade

Why Updating Controllers Requires "Protecting Success"

A well-trained robot controller can already handle many disturbances. To improve further, it needs to learn new corrective skills while also learning when not to apply them. But merely freezing the parameters of existing skills doesn't solve the "when to activate" problem—in a controller where frozen skills are routed, new entry rules can redirect states that the old skills originally succeeded on.

A study from RSS proposes a new approach: instead of constraining parameters, constrain entry conditions. At each update, the system finds a feedback skill at measured failure states and fits an entry rule that reads observations and rejects all states lying on the accumulated success trajectory. The skill is only adopted if the combined controller can autonomously reproduce the new recovery behavior and exactly replay every protected trajectory.

Skill routing is a common paradigm in hierarchical control architectures: an upper-level scheduler decides which lower-level skill to activate (e.g., walking, grasping, balance recovery) based on current observations, and each skill itself is a frozen or independently trained sub-policy. This architecture offers modularity and composability, but there is a hidden risk during updates—adding new skills or modifying scheduling rules changes how states are allocated, causing observations originally handled by old skills to be redirected to new ones. Even if the old skill parameters remain completely unchanged, a change in entry conditions can mean the old skill is no longer invoked in the states it originally excelled at, leading to behavioral degradation. Therefore, "protecting success" cannot rely solely on freezing parameters; it must also constrain the routing decisions themselves.

Core Mechanism: Constraining Entry Rather Than Parameters

The key to this method is transforming "protecting old successes" into a restriction on entry rules. Specifically, each new skill comes with an entry rule that defines a rejection region in observation space—all states on the accumulated success trajectory are rejected. This way, the new skill is only triggered near states where the old skill previously failed, avoiding interference with scenarios where the old skill succeeded.

The study provides a finite-trajectory condition for determining when this constraint preserves the prior controller's behavior. This offers a theoretical guarantee for the method's reliability.

Entry rule (initiation classifier) in robot learning typically refers to a binary classifier or gating function that takes observations (or state features) as input and outputs whether a skill is allowed to activate. Unlike directly modifying the policy network, the entry rule only affects "when to use" and not "how to use," so the modification space it constrains is smaller and more controllable. The key design of this paper is to fit the entry rule to output "reject" on the set of states along accumulated success trajectories, equivalent to carving out a local trigger region around failure states in observation space. The theoretical basis for this approach resembles safety filtering/control barrier ideas: as long as the trigger domain of the new skill is disjoint from the historical success domain, the combined system's behavior on those states matches the old controller. The finite-trajectory condition provides a sufficient criterion for this disjointness to hold under finite samples.

Experimental Validation: Four-Stream Humanoid Robot System

The study validated this mechanism on four data streams: two humanoid systems, each with two locomotion modes. Each stream underwent two updates, adding 62 recoveries on 192 held-out perturbations with zero loss of any successes from the base controller.

This means that while improving the controller's ability to handle new disturbances, the old successful behaviors were fully preserved.

Held-out perturbations refer to a set of disturbances not used during training or skill updates, reserved solely for evaluation—analogous to a test set in machine learning. Evaluating on 192 held-out perturbations tests whether the 62 new recoveries generalize to unseen conditions rather than merely memorizing the failure states observed during updates. The four data streams (two humanoid systems × two locomotion modes) form a multi-task setting used to verify the method's transferability across different morphologies and locomotion patterns. "Zero loss" means all originally successful held-out perturbations are still successfully completed after the update, which is the core metric distinguishing this method from naive fine-tuning.

The Cost of Safety

The study further quantified the cost of this guarantee. Removing the entry constraint allowed the system to recover 33 additional conditions but caused 8 protected conditions to become completely unachievable—i.e., the old skill's successes were broken. Excluding only those constraints proven necessary could not recover this cost. This shows that protecting old successes requires a certain performance trade-off, but what this cost buys is predictability and safety of behavior.

For real-world deployed robot systems, this trade-off is critical: in open environments, unpredictable degradation can have serious consequences, and sequential recovery updates offer a controllable upgrade path.

Protected conditions refer to evaluation scenarios that were successfully completed by the base controller before the update and included in the protection set. The 33 additional conditions recovered after removing the entry constraint show that the constraint indeed limits the new skill's trigger range, causing some situations that could have been resolved by the new skill to be rejected; the 8 protected condition failures show that without constraints, the new skill "usurps" states where the old skill originally succeeded. This trade-off essentially reflects the stability–plasticity dilemma: the more a system emphasizes retaining old behaviors (stability), the harder it is to absorb new capabilities (plasticity). The inability to recover the cost by excluding only "proven necessary" constraints suggests that the boundary between protection and performance is not determined by a few critical constraints but is diffuse across the entire trigger domain.

Future Outlook

This method is currently validated on humanoid robots, but its core idea—protecting success trajectories through entry rules—may apply to other learning-based control systems with skill routing architectures. Future work could explore more efficient constraint representations to reduce the performance loss incurred by protection.

One-line Summary

By constraining skill entry rules rather than parameters, recovery skills can be incrementally added to robot controllers while preserving old successful behaviors.

Paragraph Summary

The article introduces an RSS study addressing the "catastrophic forgetting" problem in incremental robot controller learning with a new idea: instead of freezing or constraining policy parameters, constrain the entry conditions of new skills. At each update, the system learns a feedback skill at measured failure states and fits an entry rule that rejects all states on the accumulated success trajectory, so the new skill is triggered only in regions where the old skill previously failed. The study provides a finite-trajectory condition as a theoretical guarantee and validates on two humanoid systems across four data streams: after two updates, 62 recoveries were added on 192 held-out perturbations with zero loss. Ablation experiments show that removing the entry constraint recovers 33 additional conditions but causes 8 protected conditions to fail, revealing the performance cost required to protect successes.