Dex-One2Many: Learning Dexterous Manipulation from a Single Human Video with Zero-Shot Real-Robot Transfer

Learning dexterous manipulation from a single human video is an appealing path toward lowering the cost of robot teaching. However, most existing methods remain at the level of "imitation"—strictly matching the action trajectories in the video, which causes rapid failure when facing new object initial poses, target poses, or grasping strategies. Another route is to use reinforcement learning (RL) to explore from scratch. While it has strong generalization potential, it struggles with high-dimensional exploration in multi-stage complex tasks. Dex-One2Many attempts to solve both problems simultaneously.

Core Idea: Abstracting Videos into Scene Graphs

The key design of Dex-One2Many is to convert a single human video into sequential scene graphs and use these graphs to guide reinforcement learning.

Scene graphs do not record precise joint angles or end-effector poses; instead, they describe relational constraints between objects. This choice of abstraction level is clever: relational constraints preserve the structural information of the task (e.g., "the tool must be picked up first, then brought into contact with the target object"), but do not fix specific numerical values, thereby leaving room for policy exploration.

Specifically, scene graphs play a role in two stages. First, as generative constraints, they guide the sampling of diverse reset states—since only relations are constrained rather than precise poses, the sampled initial states can cover object poses and grasping strategies not present in the video. Second, they provide dense rewards for each task stage, making the exploration path sufficiently short and clearly guided when the policy starts from these reset states.

This is essentially a hybrid paradigm of "structured priors + RL generalization": human videos provide the task skeleton, and RL is responsible for filling in a generalizable control policy within that skeleton.

Scene Graph is a structured representation method in computer vision and robotics. It typically uses graph nodes to represent entities (such as objects, tools, and the robot end-effector) and edges to represent relationships between entities (such as "contact," "above," and "grasping"). Compared with directly outputting dense numerical values such as joint angles or six-degree-of-freedom object poses, a scene graph describes a higher-level semantic structure. In robot manipulation tasks, the advantage of scene graphs is that relational constraints often still hold when object positions change; for example, "the cup is above the saucer" does not become invalid because their specific coordinates change. This gives scene-graph-based policies a certain natural pose invariance. Dex-One2Many further introduces the concept of "sequentialization," that is, chaining multiple scene graphs along the time axis to characterize the evolution of object relations in multi-stage tasks, thereby preserving the temporal structure of the task while avoiding overfitting to precise motion trajectories.

In reinforcement learning, dense reward is contrasted with sparse reward. Sparse reward provides signals only when the task succeeds or fails, leading to extremely low exploration efficiency in long-horizon, multi-stage tasks—the agent may receive no feedback for a long time. Dense reward provides intermediate feedback at every state or step, significantly shortening the exploration path. In manipulation tasks, dense rewards are usually designed based on distance to the goal state or completion of task stages. Dex-One2Many uses scene graphs to generate dense rewards for each task stage, essentially turning "which stage are we currently in and whether the object relations for that stage are satisfied" into a computable feedback signal. This alleviates the difficulty of pure RL exploration in high-dimensional action spaces and differs from imitation learning's frame-by-frame action matching, thus preserving policy exploration flexibility while providing guidance.

Real-to-Sim-to-Real

Dex-One2Many adopts a real-to-sim-to-real pipeline. Human videos are first parsed into scene graphs, then the policy is trained in a simulation environment, and finally transferred zero-shot to a real multi-fingered dexterous hand.

The entire training process is completed entirely in simulation, without using any real robot demonstration data. This is significant for practical deployment—it means that acquiring a new skill requires only a human manipulation video, rather than setting up a real-robot data collection pipeline.

The success of zero-shot transfer depends on the abstraction of the task essence by scene graphs. If the policy learns precise trajectories specific to the simulation environment, transfer to the real robot will inevitably fail; whereas a policy at the relational level is more likely to bridge the gap between simulation and reality.

Real-to-sim-to-real is a popular training paradigm in robot learning, referring to converting real-world data into a simulation environment, training a policy in simulation, and then deploying it back to a real robot. Its core motivation is that simulation enables large-scale parallel sampling, fast resets, and no hardware wear, while real-world data provides the true distribution of tasks and scenes. The key challenge in this pipeline is the sim-to-real gap: physical parameters, friction, and visual appearance in simulation differ from the real environment. If a policy trained directly in simulation overfits simulation details, its performance will drop sharply when transferred to the real robot. Common mitigation methods include domain randomization and task-level abstract representations. Dex-One2Many relies on the relational-level abstraction of scene graphs to bridge the gap, meaning the policy learns not specific pose values but the task structure at the relational level.

Experimental Performance: From 6.5% to 71%

The paper evaluates on five tool-use and manipulation tasks. Under seen configurations (scenes similar to those in the video), Dex-One2Many surpasses baseline methods by 6.5%.

More noteworthy is the performance under unseen scenes: its generalization ability widens the advantage gap to 71%. This number shows that the scene-graph-guided policy is not fitting the demonstrations in the video, but truly learning transferable task structure. Limited advantage under seen configurations and a large lead under unseen scenes precisely confirms the method's generalization characteristics—its value lies not in reproduction, but in handling variation.

Zero-shot refers to the ability of a model to execute directly on a test task without having seen any data for that target task during training. In the context of robot manipulation, zero-shot transfer to a real robot means the policy is trained entirely in simulation, without using any demonstrations or interaction data collected by a real robot, yet can complete manipulation on real hardware. This is one of the strictest settings for evaluating generalization ability. In contrast, few-shot fine-tuning or methods requiring real-robot data calibration often need to rebuild the data collection pipeline for each new task. Dex-One2Many achieving zero-shot real-robot transfer means the cost of acquiring a new skill is compressed to only one human manipulation video, without needing to build a real-robot teleoperation or data collection system. This property is especially important for platforms such as dexterous hands, where hardware is expensive and data collection is difficult.

Significance and Limitations

Dex-One2Many provides a balancing approach: use human videos to provide exploration priors, use RL to ensure generalization ability, and use scene graphs as the bridge between the two. This framework is especially valuable for multi-stage dexterous manipulation tasks, because the exploration space of such tasks is extremely large, and pure RL can hardly find effective policies within a reasonable time.

However, the scalability of the method still needs verification. Does the construction of scene graphs rely on manual design? How automated is the process from video to graph? These questions affect the deployment cost of the method on new tasks. In addition, the experiments cover five tasks; whether it is equally effective in more complex long-horizon manipulation still requires more evidence.

One-line Summary

Dex-One2Many generates sequential scene graphs from a single human video and guides reinforcement learning to achieve multi-stage dexterous manipulation and zero-shot real-robot generalization.

Paragraph Summary

To address poor generalization in imitation learning and difficult exploration in pure reinforcement learning for dexterous manipulation, Dex-One2Many proposes abstracting a single human video into sequential scene graphs to guide reinforcement learning. Scene graphs constrain only relations between objects rather than precise poses, so they can both serve as generative constraints to sample diverse reset states and provide dense rewards for each task stage. The method adopts a real-to-sim-to-real pipeline, with training completed entirely in simulation and zero-shot transfer to a real multi-fingered dexterous hand. On five tool-use and manipulation tasks, it surpasses baselines by 6.5% under seen configurations, and its generalization advantage expands to 71% under unseen scenes. The method offers a hybrid paradigm of "structured priors plus RL generalization," but the degree of automation in scene graph construction and scalability to longer-horizon tasks remain to be verified.