HALO: Simulation Data-Driven Zero-Shot Transfer for Humanoid Robots

Data Bottleneck: The Key Obstacle to Scaling Humanoid Robots

Humanoid robots have the potential to automate a large number of physical tasks, but the path to general autonomy is blocked by data. To map pixels and natural language to low-level joint commands, models need paired supervision signals—and such data is far from sufficient at scale. Existing methods rely on human teleoperation and wearable motion-capture devices, producing 1 hour of data for at least 1 hour of human input, with costs growing linearly with manpower, making it difficult to support general policy training.

The HALO system proposed by Apple's AI/ML research team attempts to break this bottleneck with synthetic simulation data. Its core pipeline synthesizes vision-language-motion data across 1,000 procedurally generated virtual homes, allowing data scale to expand with compute rather than being constrained by human labor. HALO acquires a range of physical skills: sitting on a chair, navigating to a specified object, turning sideways to pass through a half-open door, crouching and crawling under a table, pushing a box—and it can orchestrate these skills to solve long-horizon composite tasks. Most strikingly, despite never having seen a real home during training, HALO achieved zero-shot transfer in six unmapped real environments, with a success rate of 91% across 360 trials.

Humanoid robots have the potential to automate a large number of physical tasks, but the road to general autonomy is stuck on data. For a robot to understand an instruction like "bring me the cup on the table," the model needs to simultaneously see the visual scene, the language description, and the corresponding joint action sequence. Collecting such triplet data in the real world is extremely expensive: teleoperation requires an operator to control the robot throughout, and wearable devices require a person to wear sensors and demonstrate the motions. Either way, data production speed is limited by human time.

HALO's starting point is straightforward: if real data is too expensive, generate it with simulation. The research team built 1,000 procedurally generated virtual homes, each containing diverse furniture, objects, and layouts. In these virtual environments, the system automatically synthesizes vision-language-motion paired data, and the data volume can scale with available compute. This means training scale is no longer bottlenecked by human labor but can instead improve by piling on compute, much like training large language models.

Vision-language-motion data is a class of multimodal supervision signal in robot learning, typically consisting of visual observations (images or video), natural language instructions (such as "bring me the cup"), and corresponding motion trajectories (joint angle sequences or end-effector poses). Unlike vision-language pretraining data, it additionally requires precise physical action annotations, so it cannot be obtained directly from internet text-image pairs. Collecting this kind of data relies on teleoperation, motion capture, or simulation synthesis, and its scale and quality directly determine whether a policy can generalize to new scenarios. One of HALO's core contributions is precisely to bypass the high cost of real-world collection with procedural simulation, enabling such triplet data to be generated at scale.

How HALO Works: From Simulation to Skill Acquisition

HALO's pipeline can be understood as three layers. At the bottom is the procedurally generated virtual home, providing diverse scenes and object layouts; in the middle is automatically annotated vision-language-motion data aligning scenes, instructions, and actions; on top is policy learning, enabling the robot to extract reusable skills from these synthetic data.

According to the paper, the skills HALO acquires cover multiple aspects of everyday locomotion and manipulation: sitting on a chair requires coordinating balance with leg movements; navigating to a named object requires combining language reference with visual perception; turning sideways through a half-open door tests fine spatial reasoning; crouching and crawling under a table involves whole-body posture adjustment; and pushing a box requires force control and path planning. These skills are not isolated—HALO can combine them to complete long-horizon tasks, such as first finding a box and then pushing it to a specified location, or navigating to a target object after passing through a door.

This skill-composition capability is key to general-purpose humanoid robots. Real home environments are full of variability, and a single skill cannot handle a continuous stream of tasks. The skill library HALO learns through simulation data provides a foundation for compositional generalization.

Procedural generation refers to using algorithmic rules rather than manual modeling to create virtual scenes at scale. In HALO, researchers define a parameter space of room layouts, furniture types, object placements, and lighting conditions through code, allowing the system to automatically combine 1,000 stylistically varied virtual homes. Compared with hand-building a small number of fixed scenes, procedural generation can obtain visual and physical diversity at extremely low cost, thereby reducing a policy's overfitting to specific scenes. This approach has precedents in game development and simulation training, but HALO is a relatively representative attempt to apply it to humanoid robot vision-language-motion data synthesis.

Zero-Shot Transfer: Crossing the Simulation-to-Reality Divide

The biggest skepticism toward simulation training is always the "sim-to-real gap." HALO's answer is quite convincing: without ever having seen a real home, the system completed 360 trials in six unmapped real environments with a 91% success rate. These environments are not laboratory sets but real spaces that were not pre-mapped. Zero-shot means no fine-tuning was done for the real environments; the policy was deployed directly.

A 91% success rate is a notable number in the humanoid robotics field, especially considering that the tasks include actions with high demands on perception and motor coordination, such as passing through doors and crawling under tables. This shows that the procedurally generated virtual homes achieve a certain level of visual and physical diversity, enabling the policy to generalize to unseen real-world scenarios. Of course, the paper does not disclose the specific distribution of failure cases, and differences in success rates across task types are worth further attention.

Sim-to-real gap refers to the performance degradation that occurs when a policy trained in simulation is deployed to a real physical system due to differences in dynamics, sensor noise, lighting, and materials. Common mitigation methods include domain randomization (randomizing physical parameters and visual appearance during training) and system identification (calibrating the simulator to match the real robot). HALO's zero-shot transfer means no fine-tuning was done in the real environment and the policy was deployed directly, so the 91% success rate indicates to some extent that its procedurally generated virtual homes provide sufficient coverage in visual and physical diversity, giving the policy generalization capability for unseen real-world scenarios.

Significance and Limitations for the Industry

HALO's value lies in validating a scalable data path: replacing human collection with simulation synthesis, allowing data scale to grow with compute. If this path holds, humanoid robot training can escape the asset-heavy teleoperation model and accelerate iteration. Apple's AI/ML team has released the code and videos on the project page, providing a foundation for community reproduction and follow-up research.

A sober view is needed: the 91% success rate comes from 360 trials covering six types of environments, but the real world's long-tail scenarios go far beyond these. Whether procedurally generated virtual homes can cover the cluttered object placements, unusual lighting, and dynamic obstacles in real homes remains an open question. In addition, differences between motion data synthesized in simulation and the dynamics of real robots may be amplified in more complex contact tasks. HALO shows the direction, but there is still a distance from demonstration to large-scale deployment.

Conclusion

The core insight of HALO is that the data bottleneck for humanoid robots can be alleviated through simulation synthesis, and that synthetic data is sufficient to support zero-shot transfer to the real world. As data production shifts from being human-driven to compute-driven, the training paradigm for general-purpose humanoid robots may undergo a structural change. The next things worth watching are whether this simulation pipeline can extend to more fine-grained manipulation tasks, and whether the community can reproduce and surpass the 91% success rate on the basis of the public code.

One-line Summary

Apple's HALO trains humanoid robots with synthetic data from 1,000 virtual homes, achieving 91% zero-shot real-world transfer.

Paragraph Summary

Apple's AI/ML team proposed the HALO system, aiming to break the cost bottleneck of real data collection for humanoid robots with simulation-synthesized data. The system automatically synthesizes vision-language-motion paired data across 1,000 procedurally generated virtual homes, allowing training scale to expand with compute rather than human labor. HALO acquires skills such as sitting on a chair, navigating to named objects, turning sideways through doors, crawling under tables, and pushing boxes, and can combine them to complete long-horizon tasks. The most prominent result is zero-shot transfer: despite never having seen a real home during training, it completed 360 trials in six unmapped real environments with a 91% success rate. The article argues that this validates the feasibility of replacing human collection with simulation synthesis, but real-world long-tail scenario coverage and more complex contact tasks remain open questions.