EgoExplorer: A 100-Hour Vision-Motion-Tactile Dataset to Advance Humanoid Robot Learning

Getting humanoid robots to learn how to move and manipulate objects like humans has never been about a single action—the real challenge lies in how gait, posture, finger movements, and object contact change in sync as a task progresses. Existing datasets typically capture only one of these facets, while EgoExplorer aims to record all of them simultaneously.

Why Existing Data Falls Short

Human demonstrations are a key source for humanoid robots learning loco-manipulation, but researchers face a very concrete dilemma: to cover diverse environments, the capture device must be lightweight and wearable; yet once you prioritize portability, the number of sensors is limited, making it hard to fully capture fine joint and finger movements.

Most existing motion datasets rely on complex motion capture studios or dense markers, which are costly to collect and restricted in scenario coverage. Egocentric video datasets, on the other hand, lack full-body motion and tactile information. As a result, models often see fragmented clips where "moving" and "manipulating" are separated, rather than the intertwined process they form in real tasks. EgoExplorer's starting point is precisely to fill this gap: aligning first-person vision, reconstructed full-body motion, fine-grained finger joints, and tactile observations on the same timeline, accompanied by step-level descriptions.

loco-manipulation is a core task category in the humanoid robotics field, referring to a robot applying manipulation to objects with its hands while maintaining balance and locomotion. It differs from both traditional "fixed-base manipulator operation" and "pure legged locomotion." The difficulty lies in the fact that movement continuously changes the relative pose between the body and the object, and hand contact states and gait must be coordinated in real time—not simply walk to a position and then manipulate. Precisely because of this, the required data must simultaneously contain displacement trajectories, full-body poses, fine finger movements, and contact events. Missing any one of these would lead the model to learn "distorted" causal relationships. The reason EgoExplorer emphasizes multi-signal alignment is that the learning signals for loco-manipulation are inherently multimodal and coupled—capturing them separately would break this coupling.

A 100-Hour Wearable Observation Station

EgoExplorer is a 100-hour vision-motion-tactile observation dataset built on a wearable system, with a focus on occupational activity scenarios. Its continuous recordings align four types of signals: egocentric video, reconstructed full-body motion, finger joint movements, and tactile observations, while also providing step-level text descriptions.

The value of this design lies in preserving the complete temporal sequence of a task from start to finish, rather than static frame-sampled snapshots. For researchers working on humanoid robot retargeting and whole-body control, continuous, multimodal, semantically annotated data is more useful than mere pose sequences—models can learn the associations between "what is seen, how the body moves, how the hands manipulate, and when contact with objects occurs."

Calibrate-Complete-Constrain: Reconstructing Full-Body Motion from Sparse Sensing

Achieving this level of coverage under sparse body sensing relies on the Calibrate-Complete-Constrain pipeline proposed by the authors. Its core idea can be broken down into three steps:

Calibrate first: Before motion inference, use a reusable calibration procedure to fix the participant's body geometry and sensor placement. The key here is to decouple "mounting offset" from "true joint motion"—if the sensor placement itself has a bias, subsequent inference would mistake that bias for actual movement.

Temporal completion (Complete): Use temporal priors to fill in unobserved joint movements. Wearable devices cannot cover all degrees of freedom, and missing parts need to be inferred from the temporal continuity of motion.

Hierarchical constraints (Constrain): Through multiple layers of constraints, simultaneously promote measurement consistency, temporal continuity, and geometric consistency. The authors specifically point out that kinematic "plausibility" alone is not sufficient to guarantee these—explicit constraints are necessary.

In terms of results, compared with tracker-fitted references, the method reduces body point error by 71.9% relative to the strongest baseline; matching evaluations also show improvements in hand stability and contact estimation. This means that even with sparse sensors, reconstruction accuracy can approach the level of heavier equipment.

Retargeting refers to mapping the motion trajectory of one motion subject onto another subject with different body proportions and joint structures—in this case, transferring human motion to the joint space of a humanoid robot. Since the two differ in the number of degrees of freedom, link lengths, and joint limits, directly copying joint angles often causes the robot to self-intersect or lose balance. Therefore, it is necessary to re-solve for robot-executable joint trajectories while preserving task semantics (e.g., hand reaching a certain point, foot landing at a certain spot). The accuracy of this process depends heavily on the quality of the source data: if the human full-body motion and finger joints themselves have biases, retargeting to the robot will further amplify them. EgoExplorer's exploration of humanoid retargeting uses high-quality human demonstrations as input, then executes them in simulation via a whole-body controller, thereby validating the dataset's usability.

Preliminary Exploration: From Generation to Humanoid Retargeting

The paper also presents several preliminary studies, covering directions such as multimodal generation and conditional control, as well as humanoid robot retargeting with whole-body control in simulation.

This part is more about validating the dataset's "downstream usability": if the collected vision-motion-tactile signals can be used for conditional generation, and human motion can be effectively transferred to a simulated humanoid and executed with a whole-body controller, then the significance of this observation station goes beyond "recording"—it can provide training material for policy learning. For researchers working on human-to-humanoid transfer, the finger joint and tactile signals are particularly noteworthy—they are precisely the parts most easily sacrificed in previous datasets.

What It Means for the Field

EgoExplorer's contributions can be summarized in three points: first, it places vision, motion, touch, and finger details into the same wearable capture framework; second, it demonstrates through calibration plus constraints that sparse sensing can still yield relatively high-quality motion reconstruction; third, it directly links the dataset to humanoid retargeting, shortening the path from human demonstrations to robot policies.

Of course, the occupational activity-heavy collection bias also means the scenario distribution is specific, and the 100-hour scale is medium-sized among multimodal datasets. Whether it can later cover more everyday, messier environments will determine the upper limit of its generality. But for the goal of "letting robots learn to walk and use their hands at the same time," the direction of such multimodal aligned data is clear.

Tactile observation typically refers to force, pressure distribution, or contact event information obtained through contact sensing—direct evidence for judging whether "the hand is truly acting on an object." In datasets relying only on vision, contact can often only be inferred indirectly from occlusion relationships, which easily fails during fast movements or when the hand is self-occluded. After aligning tactile signals with finger joint movements, models can distinguish between "finger moves but does not touch an object" and "finger contacts and applies force"—two visually similar states. This is crucial for learning operations such as grasping, pressing, and assembly. Tactile data collection usually relies on wearable sensing gloves, and the weight and number of the devices themselves in turn affect the wearer's natural movements—this is also one of the practical reasons why EgoExplorer adopts sparse sensing and requires calibration and constraint pipelines.

One-line Summary

EgoExplorer uses 100 hours of wearable data to align vision, motion, fingers, and touch, helping humanoid robots learn loco-manipulation.

Paragraph Summary

EgoExplorer is a 100-hour wearable vision-motion-tactile dataset for humanoid robot loco-manipulation learning, designed to address the problem that existing data records only a single modality and cannot capture the synchronized changes of gait and manipulation within a task. It aligns first-person video, reconstructed full-body motion, finger joint movements, and tactile observations on the same timeline, with step-level text descriptions. To achieve this coverage under sparse sensing, the authors propose the Calibrate-Complete-Constrain pipeline: first calibrate sensor wearing offsets, then use temporal priors to complete unobserved joints, and finally apply hierarchical constraints to ensure measurement, temporal, and geometric consistency, reducing body point error by 71.9% relative to the strongest baseline. The paper also preliminarily validates multimodal generation, conditional control, and humanoid retargeting with whole-body control in simulation.