VioLA: Training Generalist Humanoid Robot Control Policies with Human Data

The Two Major Challenges of Humanoid Robots: Action Space and Data Scarcity

To make a humanoid robot respond to commands with its whole body, researchers have long faced two obstacles. First, the humanoid action space is vast and highly coupled—legs, arms, and fingers must move in coordination while maintaining balance, making joint-level motion learning exceptionally difficult. Second, demonstration data for humanoid robots is extremely scarce. Existing generalist policies cannot execute new commands out of the box; they must first be fine-tuned on teleoperation demonstration data for each task before deployment.

By contrast, human motion demonstration data is far more abundant. But the problem is that human motion recordings are not robot commands—the two do not share an action space. How to turn massive amounts of human motion data into usable training signals for humanoid robots is the core breakthrough of this research.

Teleoperation demonstration data refers to data in which a human operator controls a robot in real time through motion capture equipment, exoskeletons, or remote controllers to complete tasks, while synchronously recording joint trajectories and sensor readings. It has long been the mainstream data source for robot imitation learning: high quality, naturally aligned with the robot's own action space, and directly usable for policy fine-tuning. But its bottleneck is also prominent—every frame of data requires a real operation, collecting a single multi-step task often takes tens of minutes, and it is hard to scale. The higher the dimensionality of the robot action space (humanoid robots often have dozens of degrees of freedom) and the more complex the task, the higher the collection cost. This explains why the humanoid robotics field has long lacked training corpora at internet scale, and it constitutes the core constraint that this paper attempts to circumvent.

Generalist policy refers to a robot policy model pretrained on multiple tasks and scenarios that can be transferred to new commands, typically based on a vision-language-action (VLA) architecture. Unlike specialized policies trained for a single task, generalist policies pursue "out-of-the-box" generalization: given a new language command and visual input, directly output actions without retraining. However, in the robotics field, constrained by heterogeneous action spaces and scarce demonstration data, existing generalist policies can often only achieve "fine-tuning per task after pretraining," and true zero-shot generalization remains an open problem. VioLA redefines the prediction target of generalist policies precisely against this backdrop.

VioLA's Core Idea: Predicting Motion Latents Instead of Joint Commands

VioLA's approach is to change what the generalist policy predicts. Instead of directly outputting joint commands, it predicts motion latents for the body and hands. These latents are executed on the robot by pretrained body and hand controllers, while corresponding motion encoders map human motions and robot motions into the same latent space.

The direct benefit of this design is that a segment of human video is naturally labeled in the policy's action space. Human motions and robot motions share a latent space representation, meaning human demonstrations can directly serve as training data, without the need to collect teleoperation data separately for each task. The training demonstration pool thus expands to 140.6 million frames, of which 93.2% come from humans. The change in data scale and source structure forms the foundation of VioLA's generalization capability.

Motion latents refer to encoding the high-level semantics of a motion segment with low-dimensional continuous vectors, rather than raw per-joint control quantities. They are typically extracted from motion sequences by an encoder and then restored to executable commands by a decoder or controller. Compared with high-dimensional, strongly coupled representations like joint angles, latents decouple "what motion to perform" from "how exactly to drive the joints": the former is learned by the policy, the latter is delegated to an already-trained controller. This both reduces the dimensionality of the action space the policy needs to model and makes motions from different sources (human, robot) comparable in a shared representation. VioLA's key is precisely to change the policy output from joint commands to such latents.

Motion encoder and action latent space alignment: The motion encoder's job is to map motion sequences from different sources—human motion capture or video recordings, robot joint trajectories—into the same latent vector space. Only when human motions and robot motions land in comparable positions in the latent space does a human video become equivalent to "data already labeled in the policy's action space." This cross-embodiment alignment is the theoretical premise for VioLA's reuse of human data: if the encoder's alignment quality is insufficient, motion distortion will appear when transferred to the robot, so the joint training quality of the encoder and pretrained controller directly determines the upper bound of final performance.

Zero-Shot Performance: Substantially Surpassing Existing Generalist Policies

VioLA's most striking result is its zero-shot performance on real robots. Without any task-specific fine-tuning, it can execute locomotion commands on a real robot with a 100% success rate; by comparison, GR00T N1.7 achieves 16.7%, and Ψ₀ achieves 0%.

On manipulation tasks, VioLA likewise achieves an 88.6% success rate without task fine-tuning. Moreover, the method reproduces the same results on two VLA backbones and one world-action model backbone, indicating that its approach does not depend on any specific model architecture. The research team also validated a more radical setting: a generalist policy trained solely on human demonstrations can complete locomotion tasks zero-shot on a real robot. This directly addresses the bottleneck of "scarce humanoid demonstrations"—human data itself is a sufficient training source.

Zero-shot in robot learning specifically means the model directly executes commands in a new real environment without having seen any demonstration of that task and without any task-specific fine-tuning. This is fundamentally different from "few-shot fine-tuning": the latter allows adjusting parameters with a small number of demonstrations, while zero-shot requires the policy to rely entirely on generalization capability acquired during pretraining. Zero-shot success rate is therefore regarded as one of the strictest metrics for measuring the true generalization level of a generalist policy. VioLA's reported 100% locomotion and 88.6% manipulation success rates were both achieved on real robots without any task fine-tuning, which is also the significance of its comparison with GR00T N1.7 and Ψ₀—the gap reflects a difference in the generalization mechanism itself, not a difference in the amount of fine-tuning data.

Backbone refers to the underlying neural network architecture that supports a VLA or world-action model, typically responsible for fusing vision, language, and state inputs and outputting action representations. Different backbones vary considerably in parameter scale, attention structure, and training recipe. VioLA achieves consistent results across two VLA backbones and one world-action model backbone, indicating that its gains mainly come from the interface design of "predicting motion latents" rather than incidental benefits from a particular network structure, which is also important evidence of its methodological transferability.

Why This Approach Is Worth Attention

VioLA's value is not in beating some benchmark number, but in redefining the input-output interface of generalist humanoid policies. The traditional route addresses the two problems of "action space coupling" and "scarce demonstration data" by separately resorting to hierarchical control and large-scale teleoperation collection; VioLA handles both problems together through latent space alignment—latents both reduce the coupling difficulty of joint-level learning and allow human data to be directly reused.

From an engineering perspective, the combination of a pretrained controller plus a motion encoder also has good reusability: as long as a certain type of motion can be mapped into the same latent space, similar data-reuse logic may transfer to other embodiments or tasks. Of course, this method still depends on the alignment quality between the pretrained controller and the encoder, and whether the latent space's representational capacity is sufficient to cover more complex whole-body coordination tasks remains to be verified. Code and checkpoints will be publicly released, providing conditions for community reproduction and further exploration.

One-line Summary

VioLA reuses human motion data by predicting motion latents, enabling zero-shot generalization for humanoid robots.

Paragraph Summary

To address the two major challenges of humanoid robots—a vast, coupled action space and scarce demonstration data—VioLA proposes changing the prediction target of generalist policies: instead of directly outputting joint commands, it predicts motion latents for the body and hands, executed by pretrained controllers, and aligns human and robot motions into the same latent space via motion encoders. Human videos can thus directly serve as training data, expanding the demonstration pool to 140.6 million frames, 93.2% of which come from humans. In real-robot zero-shot evaluations, VioLA achieves a 100% locomotion success rate, far surpassing GR00T N1.7's 16.7% and Ψ₀'s 0%, and an 88.6% zero-shot success rate on manipulation tasks, reproducible across multiple backbones. The researchers believe the method handles both bottlenecks together through latent space alignment, but the latent space's representational capacity for complex whole-body coordination tasks still needs verification.