New Breakthrough in Dexterous Hands: Achieving In-Hand Object Pose Reach and Safety Filtering

Task Definition and Challenges

In-hand object pose reaching requires the robot to adjust an object to a specified position and orientation (position error within 2 cm, orientation error within 0.4 rad) through multi-finger coordination, and to handle multiple consecutive targets rather than a single motion. This is far more complex than simple grasping or rotation: it requires diverse and difficult grasps while keeping the object stable throughout. The research team notes that as manipulation difficulty increases, the risk of dropping the object rises significantly, yet prior work has rarely measured safety systematically.

Training Recipe: Zero-Shot Transfer from Simulation to Reality

This work proposes a training pipeline that requires no human demonstration data, with core ideas including:

• Diverse resets and target generation: By generating a grasp dataset, the policy is exposed to rich initial states and target poses during training, avoiding overfitting to a few grasp modes.
• Adaptive friction curriculum: The friction coefficient is dynamically adjusted during training so the policy gradually adapts to different contact conditions, enhancing robustness.
• Careful system identification: By calibrating the gap between simulation and the real system, the policy can be transferred zero-shot to the real LEAP hand, requiring only motion capture or a single wrist-mounted depth camera as input.

This recipe enables the state-based policy to perform well on the real robot: over more than 13.3 hours of real-world evaluation covering 4 objects, 4800 unseen targets, and diverse wrist orientations, the policy reached 95.2% of targets (within 2 cm position error and 0.4 rad orientation error).

Zero-shot Transfer refers to deploying a policy trained in simulation directly onto a real robot without any real-world fine-tuning or additional data collection. This concept is crucial in robot learning because real-world data collection is costly, slow, and risks hardware wear. The key to achieving zero-shot transfer lies in closing the "sim-to-real" gap, typically relying on domain randomization (randomizing physical parameters such as mass, friction, and visual textures during training) and system identification (calibrating the simulator to better match the real system). In this paper, the research team uses careful system identification and an adaptive friction curriculum to enable the policy to directly achieve a 95.2% success rate on the real LEAP hand without any real-world samples.

Adaptive Friction Curriculum is a curriculum learning strategy that dynamically adjusts the friction coefficient in the simulated environment during training, allowing the policy to gradually adapt to different contact conditions from easy to hard. The core idea is: if extreme friction values are used from the start, the policy may struggle to learn stable grasps; by expanding the friction range in a curriculum manner, the policy can gradually learn to remain robust under both low friction (prone to slipping) and high friction (prone to sticking). This approach is especially suitable for multi-finger in-hand manipulation, because the contact force distribution between fingers and object is complex, and friction uncertainty is one of the main causes of object drops. In this paper, this curriculum significantly improves the policy's generalization ability in real environments.

Safety Filter: Adjustable Conservatism

To reduce the risk of dropping objects, the research team adds a learned residual safety filter on top of the task policy, whose conservatism is adjustable. The filter superimposes a correction on the action output, tending toward more conservative behavior when potential danger is predicted. Experiments show that the filter reduces drops by 75% while still completing 91.5% of targets—a small performance drop. This adjustable design allows trading off performance and safety according to the actual application scenario.

Residual Safety Filter is a safety mechanism that superimposes a correction on top of the base policy's output. Specifically, the base policy (task policy) generates actions to accomplish the goal, while the safety filter learns a residual correction that pushes actions in a more conservative direction when potential danger (e.g., the object is about to slip) is predicted. The filter is typically trained with safety as the optimization objective, and its conservatism can be controlled by adjusting parameters—the more conservative, the lower the drop risk, but possibly at the cost of some task success rate. In this paper, the filter reduces drops by 75% while maintaining a 91.5% target completion rate, demonstrating an adjustable trade-off between performance and safety.

Generalization Ability and Portable Skills

Beyond basic pose reaching, these policies exhibit several impressive capabilities:

• They can rotate a light bulb held in an arbitrary pose;
• They can reach out-of-distribution targets;
• They can withstand rapid robotic arm motion and external disturbances;
• They can be used as a portable skill with only a wrist camera, such as screwing in a light bulb.

These results indicate that the policy is not only highly adaptable but also has the potential for deployment in real environments. The research team releases the full recipe as a platform to promote research on performance, reliability, and safety in high-dimensional dexterous systems.

Out-of-Distribution Targets refer to target poses or object states encountered during testing that have not appeared in the training data, or whose distribution differs significantly from the training distribution. In robot learning, policies easily overfit to targets seen during training, leading to a sharp performance drop in out-of-distribution scenarios. In this paper, the policy's ability to reach out-of-distribution targets indicates that the representations it learns have some generalization ability rather than simply memorizing training samples. This benefits from the diverse resets and target generation mechanisms during training, which expose the policy to rich initial states and target poses, thereby enhancing its adaptability to external changes.

Significance and Outlook

The core contribution of this work is incorporating safety into the evaluation framework of dexterous manipulation and providing a reproducible training scheme. Through simulation training and zero-shot transfer, it lowers the cost of data collection; through the safety filter, it makes the trade-off between performance and safety controllable. For teams interested in studying high-dimensional dexterous systems, this provides a valuable starting point. In the future, how to extend this approach to more object types and more complex tasks remains a direction worth exploring.

One-line Summary

This research proposes an in-hand object pose reaching task, uses a simulation training recipe to achieve zero-shot transfer, and introduces an adjustable safety filter to balance performance and safety.

Paragraph Summary

This work focuses on the in-hand object pose reaching task for multi-finger dexterous hands, requiring the robot to move an object to almost any position and orientation within finger reach, and to complete long sequences of targets consecutively under arbitrary wrist orientations. The research team develops a simulation training recipe for the LEAP hand that requires no human data, including a diverse grasp dataset, an adaptive friction curriculum, and careful system identification, achieving zero-shot transfer to the real hand. The state-based policy achieves a 95.2% target success rate over more than 13.3 hours of real-world evaluation. In addition, a learned residual safety filter with adjustable conservatism reduces object drops by 75% while maintaining a 91.5% target completion rate. The policy also demonstrates generalization ability, can rotate a light bulb in an arbitrary pose, withstand disturbances, and serve as a portable skill using only a wrist camera.