REFACTOR-VLA: Enabling Robots to Autonomously Build Reusable Skill Libraries

From "Monolithic Models" to "Skill Libraries": A Paradigm Shift in Robot Learning

Today's mainstream Vision-Language-Action (VLA) models — including OpenVLA, π0, RT-2, and RDT-1B — are all so-called "monolithic" models. Their approach is straightforward: take visual and language inputs, then directly generate raw motion commands or very short action sequences.

This design may seem clean and efficient, but it hides a fundamental flaw: the model never organizes behavior into reusable, well-defined abstractions. In other words, the robot effectively "starts from scratch" every time it executes a task, rather than drawing on already-mastered skill modules like "grasp," "pour," or "tighten" — the way a human would.

This flaw produces two notable consequences. First, long-horizon task performance suffers — complex tasks requiring multiple coherent steps are often beyond a monolithic model's reliable reach. Second, interpretability is poor — it's difficult for researchers to understand what the model has actually learned, and the decision-making process is nearly a black box.

Vision-Language-Action (VLA) models have become the dominant framework in embodied AI in recent years. They typically use a pretrained visual encoder (such as CLIP or ViT) to process camera images, a large language model to handle natural language instructions, and an action head to output low-level control signals such as joint angles, end-effector poses, or velocity commands. RT-2, proposed by Google DeepMind in 2023, is a landmark work that transfers web-scale vision-language pretraining to robot control; π0 comes from Physical Intelligence and focuses on general dexterous manipulation; RDT-1B and OpenVLA are large-scale open-source VLA efforts from academia. What these models share is a "perceive-then-act" philosophy — going directly from raw perception to raw action, with no explicit behavioral abstraction layer in between. This end-to-end design excels on simple, short-horizon tasks, but its limitations become increasingly apparent as task complexity and step count grow.

The Core Challenge: Determining When Two Actions Are "Behaviorally Equivalent"

For a robot to reuse skills like building blocks, one seemingly simple but deeply thorny problem must first be solved: how do you determine whether two action sequences are behaviorally equivalent?

This is precisely the core question that most existing skill-discovery methods sidestep. Approaches like AtomicVLA and AtomSkill, for instance, attempt to group and categorize action sequences, but they tend to avoid directly assessing "behavioral equivalence," instead relying on surface-level similarity metrics or heuristic rules.

Why Judging "Behavioral Equivalence" Is So Hard

The complexity lies in the fact that two numerically distinct action trajectories may functionally accomplish the same thing, while two trajectories that look similar may correspond to entirely different task intentions.

• Context dependence: The same arm movement can carry completely different semantics depending on the object and scene.
• Granularity ambiguity: Where exactly does a "skill" begin and end? Is it a single joint motion, or an entire coordinated sequence?
• Unsupervised constraints: Without manual annotation, the model must figure out on its own which behavioral segments are worth abstracting into a reusable unit.

Without a reliable way to judge behavioral equivalence, any so-called "skill library" is just a pile of redundant, fragmented action snippets that don't meaningfully improve generalization or reusability.

In robot learning, "skill discovery" has been an active research topic for several years, typically tied to Hierarchical Reinforcement Learning (HRL). Classic approaches like the Options Framework decompose policy into a high-level "option selection" layer and a low-level "option execution" layer. Methods like DIAYN and VALOR attempt to unsupervisedly surface skills by maximizing information-theoretic objectives that encourage skill diversity. However, most of these approaches are validated in simulation or low-dimensional state spaces, and transferring them to real-world robotic manipulation faces challenges like high-dimensional observations and noisy dynamics. More recent works such as AtomicVLA and AtomSkill attempt to integrate skill discovery with VLA frameworks, but as noted above, they still primarily rely on surface-level clustering of action sequences — such as temporal distance or keyframe detection — rather than deep modeling of semantic equivalence in the sense of "doing the same thing."

REFACTOR-VLA's Core Approach: Typed Motor Programs

The central innovation of REFACTOR-VLA is proposing Unsupervised Library Learning, anchored by the key concept of "Typed Motor Programs."

The notion of "typing" draws on the idea of type systems in programming languages. In programming, types help us distinguish "integers," "strings," and "functions," and ensure they are combined correctly. REFACTOR-VLA attempts to introduce similar structure into robot behavior learning — distilling motor behaviors into reusable program modules with explicit types.

Three Key Advantages of Typed Motor Programs

1. Improved long-horizon task capability: When a robot has a structured skill library, it can compose skills like calling functions to complete multi-step complex tasks, without relearning from scratch at the raw instruction level each time.
2. Enhanced interpretability: Typed program modules allow researchers to clearly see "what skills the robot has learned" and "how it combines them," breaking open the black-box nature of monolithic models.
3. Generality of unsupervised learning: By not relying on large-scale human annotation, this approach has the potential to scale across broader, cheaper data sources, resulting in stronger scalability.

The name "REFACTOR" is itself meaningful — it echoes the software engineering concept of "refactoring," which means improving internal structure and extracting reusable components without changing external behavior. What REFACTOR-VLA does is precisely to "refactor" a robot's raw action stream: distilling structured, reusable skill abstractions from a noisy flow of motion commands.

The concept of "typed motor programs" fuses ideas from two fields: Program Synthesis and Motor Primitives. Program synthesis has long studied how to automatically derive structured, composable programs from examples; motor primitives are the traditional robotics way of describing fundamental motion patterns, with Dynamic Movement Primitives (DMP) being a classic representative. The key significance of introducing a "type system" into motor programs is that types not only describe what a behavior is, but also constrain what it can be combined with. For example, a skill typed as "grasp(object) → held-state" must produce an output state type compatible with the input type of the next skill — structurally guaranteeing semantic validity of skill composition, rather than relying solely on temporal concatenation. This approach intersects with neuro-symbolic methods, seeking to bridge the gap between neural networks' perceptual power and symbolic systems' compositional generalization ability.

Significance and Outlook: From End-to-End Generation to Structured Skill Composition

From a broader perspective, REFACTOR-VLA represents an important research direction in embodied AI: moving from "end-to-end raw action generation" toward "structured skill abstraction and composition."

This parallels the evolution in large language models from "token-by-token generation" to "tool use and modular reasoning." When a model can organize knowledge into reusable abstract units, its generalization capability, interpretability, and ability to handle complex tasks all take a leap forward.

Of course, this direction still faces many challenges: Is the criterion for behavioral equivalence robust enough? Can typed programs be reliably reused under the noise and uncertainty of real physical environments? How do we evaluate and validate the quality of an unsupervisedly learned skill library? These are questions that future research will need to answer.

Regardless, REFACTOR-VLA directly confronts the core problem that previous methods have deliberately avoided, offering a more structured and interpretable path for robot skill learning. For researchers and practitioners hoping to enable robots to genuinely tackle complex, long-horizon tasks, this direction is well worth continued attention.

Note: This article is based on analysis of publicly available abstract information from the research. Complete experimental data and technical details await full disclosure in the original paper.