HiPHI Benchmark: A High-Precision Human Motion and Object Interaction Dataset for Humanoid Robot Learning

Humanoid robot learning has long been trapped in a fundamental contradiction: algorithm iteration progresses far faster than the accumulation of high-quality motion data available for training. Internet videos appear to be an inexhaustible library of motion material, but they are nearly unusable in terms of physical precision and 3D annotation. Existing motion capture datasets, while relatively high in precision, generally lack synchronized records of object interaction and cover only a narrow range of action types. This data gap directly constrains the ability of Embodied AI and Physical AI to move from simulation to reality.

Why Existing Data Isn't Enough

Human motion in internet videos suffers from three fatal flaws. First, the viewpoint is single and heavily occluded, making it impossible to reconstruct complete 3D skeletal motion. Second, force and contact information is missing, so robots cannot learn the dynamic constraints needed for tasks like pushing, pulling, and carrying. Third, action labels are primarily semantic categories, leaving a huge gap between them and the low-level commands required for robot control, such as joint angles and end-effector trajectories.

The problems with existing motion capture datasets are more subtle. They are typically collected in controlled laboratory environments, with action types concentrated on whole-body movements without object involvement, such as walking, jumping, and turning. Even when some datasets include object interaction, the object trajectories and human motion often lack frame-level synchronization, making it difficult for trained policies to establish a joint "human-object" representation.

HiPHI's starting point is precisely to address the shortcomings in both directions simultaneously: using a linguistic action classification system to guide data collection, ensuring systematic coverage of action types; and using synchronized object trajectories and mesh data to make human-object interaction a learnable, transferable training signal.

Motion Capture is a technology that records the 3D motion trajectories of human joints through optical markers, inertial sensors, or vision algorithms, typically outputting skeletal hierarchy angle sequences or end-effector poses. Traditional optical motion capture relies on multi-camera arrays and reflective markers, achieving millimeter-level precision, but must be conducted in controlled laboratories, and marker occlusion directly causes data loss. Inertial motion capture is portable but suffers from cumulative drift. This precision-scenario trade-off is precisely the technical root cause of the narrow action types in existing datasets: laboratory environments struggle to set up rich object interaction scenarios, while field collection sacrifices 3D precision.

Sim-to-Real Transfer refers to the process of directly deploying policies trained in simulation environments onto real hardware. Its core obstacle is the physical discrepancy between simulation and reality, including friction coefficients, contact stiffness, sensor noise, and actuator latency. The higher the physical precision of motion data and the more accurate the object interaction annotations, the closer the dynamic constraints learned by the policy in simulation are to reality, and the higher the transfer success rate. Conversely, if the 3D skeletal precision of training data is insufficient or force contact information is missing, the policy will learn spurious dynamic correlations and quickly fail on real hardware.

How FrameNet Guides Action Collection

FrameNet is a linguistic framework for describing human actions. It organizes actions by semantic frames, where each frame contains participant roles, manner of action, and typical scenarios. The HiPHI team introduced this framework into the design of motion capture collection, replacing the traditional "action category checklist" with frame coverage, thereby systematically covering a broad action space from whole-body movement to fine manipulation.

The value of this methodology lies in its scalability. The collection plans of traditional datasets rely on researchers' intuitive enumeration, which easily leads to repeated collection of popular actions and omission of niche but important actions. FrameNet's frame system provides an exhaustive "action map," and collection work can be filled in frame by frame, significantly reducing blind spots in data coverage.

FrameNet was initiated by UC Berkeley in 1997 and is a representative resource of frame semantics in computational linguistics. Its core unit is the "Semantic Frame," where each frame describes a type of event or state and annotates the roles involved, such as agent, patient, instrument, and result. For example, the "Carrying" frame includes roles such as carrier, carried object, source, and goal. FrameNet has covered over a thousand frames and hundreds of thousands of annotated example sentences, and is widely used in semantic role labeling, event extraction, and question answering systems. Introducing it into action collection design essentially transforms the action space from a "discrete category checklist" into a "frame graph with role constraints," enabling collection plans to exhaustively enumerate interaction scenarios by role combinations, thereby systematically avoiding omissions.

Why Object Trajectories and Meshes Are Critical

With only skeleton motion data, a robot can learn to "move like a human," but cannot learn to "do things like a human." The core of tasks like carrying, pushing, and pulling lies in the force transmission and spatial relationships between the human body and objects. HiPHI provides synchronized object trajectories and 3D meshes for each interaction action, so that training data simultaneously contains "how the human moves" and "how the object moves."

This synchronized annotation enables reinforcement learning policies to establish a joint state representation of human and object. The policy network no longer observes only joint angles, but simultaneously perceives the object's position, orientation, and motion trends, thereby learning complex coordinated behaviors in simulation such as "grasping the handle and adjusting body center of mass to push a heavy object."

Scaling Effects and Sim-to-Real Transfer

According to the whitepaper, reinforcement learning policies trained on HiPHI data exhibit clear scaling effects: policy performance continues to improve as data volume increases, with no apparent saturation. This characteristic is particularly important for data-driven humanoid robot control, as it means that continuing to expand collection scale remains a worthwhile investment.

The more critical validation lies in sim-to-real transfer. The research team deployed policies trained in simulation onto real humanoid robots, validating transferability from virtual environments to physical hardware. Although the whitepaper does not disclose specific transfer success rate data, this validation itself demonstrates the usability of HiPHI data in terms of physical precision—if the 3D precision of motion data is insufficient or object interaction annotations contain systematic biases, sim-to-real transfer would be nearly impossible.

Reinforcement Learning learns control behaviors by having an agent interact with the environment and adjust its policy based on reward signals. In humanoid robot control, the state space typically includes joint angles, angular velocities, center of mass position, and object poses, while the action space consists of target torques or position commands for each joint. Unlike supervised learning, reinforcement learning does not rely on expert-annotated action labels, but instead optimizes cumulative returns through trial and error, making it particularly suitable for leveraging large-scale parallel sampling in simulation environments. However, this also means the policy is highly dependent on the physical fidelity of the simulator: contact models, friction cone approximations, and solver step sizes all affect whether the learned policy can transfer to real hardware. The synchronized object trajectories and meshes provided by HiPHI enable reward functions and state representations to simultaneously encode the coupled dynamics of human and object, which pure skeleton data cannot support.

What It Means for Humanoid Robot Learning

The emergence of HiPHI reflects a broader trend: competition in embodied AI is shifting from model architecture to data infrastructure. As the Scaling Law of large language models has been widely validated, the problem facing the robotics field is not "whether the model is big enough," but "whether the data is good enough and plentiful enough." Whoever masters systematic, high-precision, scalable motion data collection capability masters the underlying barrier to humanoid robot learning.

From an engineering perspective, HiPHI's FrameNet-guided collection method provides a reusable template for data production pipelines. This methodology is not limited to any specific robot platform and can theoretically be adapted to different motion capture systems and robot morphologies. If this model is adopted by more teams, the fragmentation problem of motion datasets is expected to be alleviated, and cross-dataset policy transfer will become more feasible.

One-line Summary

HiPHI uses FrameNet to guide collection of human-object interaction motion data, validating scaling effects and sim-to-real transfer.

Paragraph Summary

Humanoid robot learning has long been constrained by the scarcity of high-quality motion data: internet videos lack 3D precision and force contact information, while existing motion capture datasets lack synchronized object interaction records and have narrow action coverage. HiPHI proposes using the FrameNet linguistic framework to systematically guide motion collection, replacing subjective enumeration with frame coverage to avoid action type blind spots; simultaneously, it provides synchronized object trajectories and 3D meshes for each interaction action, enabling reinforcement learning policies to establish joint state representations of human and object. According to the whitepaper, policies trained on HiPHI exhibit clear scaling effects, with performance continuing to improve as data volume increases, and have successfully completed policy transfer from simulation to real humanoid robots. The article argues that this marks a shift in embodied AI competition from model architecture to data infrastructure, where systematic, high-precision, scalable collection capability will become the underlying barrier.