APAC Dataset: First-Person Stereo Vision Robot Training Data for Real-World Work Environments

Why Existing Egocentric Datasets Fall Short

In the fields of robot learning and Embodied AI, egocentric video datasets have long been a crucial resource for training models to understand human behavior. However, the most well-known egocentric datasets currently available—such as Epic-Kitchens—almost exclusively focus on a single scenario: people cooking in their home kitchens.

Epic-Kitchens is one of the largest egocentric video datasets available today, containing hundreds of hours of kitchen activity footage. Similar datasets include Meta's Ego4D (covering daily life recordings from multiple locations worldwide) and EGTEA Gaze+ (a cooking dataset combined with eye-tracking). While these datasets have been instrumental in advancing action recognition and video understanding, they share a fundamental limitation: the domain shift problem caused by scene homogeneity. In machine learning, when the distribution of training data significantly differs from the deployment environment's distribution, model performance degrades dramatically—this is the well-known "out-of-distribution generalization" challenge.

This creates an obvious mismatch problem. As the original source points out: the robots we're training with this "kitchen data" are ultimately destined to work in warehouses, garages, and factory floors. Grasping, chopping, and stirring actions from cooking scenarios are fundamentally different from assembly, inspection, and material handling in industrial environments—not only do the action patterns differ, but so do object categories, lighting conditions, and spatial layouts. This distributional bias directly limits models' generalization ability in real work scenarios.

The APAC Egocentric Stereo dataset was created precisely to fill this gap. It shifts the camera from the kitchen to actual "work sites," aiming to provide training material that more closely matches the real deployment environments for industrial and service robots.

Core Composition and Scene Coverage of the APAC Dataset

APAC Egocentric Stereo contains 12 first-person recorded videos documenting people actually performing their jobs. The covered scenes include:

• Automotive garage
• Construction site
• Electronics factory
• Bar
• Shipment hub
• Laundromat

What these scenes have in common is that they're all places where robots will actually need to "work" in the future, involving extensive fine manipulation, tool use, and environmental interaction. The operational complexity and environmental dynamics of each scene far exceed those of a static home kitchen—for example, automotive repair involves using multiple specialized tools in confined spaces, construction sites feature highly unstructured environments and safety constraints, and electronics factories demand extremely high operational precision.

Detailed Multi-Modal Annotation Specifications

From a technical specification standpoint, this dataset's capture configuration is quite robust:

• Head-mounted stereo camera setup: Each "eye" records at 1920×1080 resolution and 30 fps, forming binocular stereo vision
• Depth render: Provides depth information for the scene
• Hand and head tracking: Records the wearer's motion and pose
• Action captions: Text descriptions of what the wearer is doing at each moment

The entire dataset is segmented into 248 clips, covering 62 different verbs. These verbs represent various core actions in work scenarios, with diversity far exceeding that of a single cooking scene, providing a foundation for models to learn a broader range of operational skills. For comparison, typical kitchen datasets usually contain no more than 30 core verbs, highly concentrated on cooking-related actions like "cut," "grab," "place," and "pour," while APAC's 62 verbs cover assembly, inspection, transport, sorting, and other industrial-grade operations.

The Value of Stereo Vision and Depth Information for Robot Training

Particular attention should be paid to the fact that APAC uses stereo capture rather than a monocular camera. For robot training, this choice is highly significant.

Binocular stereo vision simulates the parallax principle of human eyes, using two cameras separated by a certain baseline distance to simultaneously capture the same scene. It uses pixel offsets (disparity) between corresponding points in left and right images to triangulate and calculate object depth information. Compared to monocular depth estimation, which relies on learned priors to infer depth, stereo vision provides geometrically more accurate and reliable depth information.

Binocular vision combined with depth rendering allows models to directly learn three-dimensional spatial relationships—precisely the capability robots need to perform grasping, placing, obstacle avoidance, and other operations. Compared to monocular data that can only infer 2D planar information, stereo data is closer to the perceptual dimensions robots need in the physical world. For robotic manipulation, precise depth perception directly determines the success rate of grasp planning—robots need to know the exact distance from target objects to the end effector, the 3D shape of objects, and the spatial distribution of surrounding obstacles. The APAC dataset provides both raw stereo image pairs and pre-computed depth renders, allowing researchers to either train end-to-end depth estimation models or directly use depth information as input for manipulation policies.

Furthermore, hand and head tracking data provides action-level supervision signals for imitation learning. Imitation learning is one of the core paradigms in robot learning, with the fundamental idea of learning policies by observing expert (usually human) demonstrations, rather than through trial-and-error reinforcement learning. In imitation learning, the quality and information richness of demonstration data directly determines learning outcomes.

Hand tracking data provides 6-DoF hand pose information—including 3D position (x, y, z) and 3D rotation (roll, pitch, yaw)—which can be directly mapped to target trajectories for robot end effectors. Head tracking reveals human attention allocation patterns, i.e., where operators are looking while performing tasks, which is valuable for training robots' active perception and gaze strategies. Researchers can see not only "what the person is doing" but also precisely "how the hands and head are moving," which is critical for transferring human demonstrations to robot policies. Combined with semantic information from action captions, researchers can build language-conditioned imitation learning systems that enable robots to execute operations based on natural language instructions.

Using FiftyOne for Multi-Stream Synchronized Browsing and Data Exploration

A dataset's usability often determines its actual value. The APAC dataset has been parsed into FiftyOne format, an open-source tool specifically designed for visualizing and managing machine learning datasets.

FiftyOne, developed by Voxel51, focuses on solving problems related to visualization, annotation quality checking, data filtering, and model evaluation for computer vision and multi-modal datasets. In traditional workflows, researchers often need to write extensive scripts to parse, align, and browse multi-modal data—a process that is both time-consuming and error-prone. FiftyOne simplifies this through a unified data model and interactive interface—users can perform programmatic data filtering and analysis via the Python API while leveraging a web interface for visual exploration.

In FiftyOne, all data streams scroll on a single shared timeline: left and right eye views, depth maps, tracking data, and captions are all presented in sync. Users can load the entire dataset with a single line of code, greatly lowering the barrier to exploration and analysis.

For researchers, this multi-stream synchronized browsing is extremely user-friendly—allowing cross-validation of visual, depth, and annotation alignment within the same interface, and quick location of segments of interest. This level of toolchain maturity is the critical bridge that takes a dataset from "published" to "widely adopted." If a dataset requires researchers to spend days writing data loading and visualization code, its actual adoption rate will be significantly diminished.

Quick Start Options

The dataset provides two access methods:

1. Hugging Face dataset page: Voxel51/APAC-Egocentric-Stereo, suitable for downloading and local use
2. Online Space browser: harpreetsahota/APAC-Egocentric-Stereo-Explorer, explore directly in the browser without deployment

Significance of the APAC Dataset for Embodied AI Research

The release of APAC Egocentric Stereo reflects an emerging consensus in the Embodied AI field: the scene relevance of data directly determines a model's real-world effectiveness.

Embodied AI emphasizes that intelligent agents must learn and understand the world through physical interaction with their environment, rather than relying solely on text and images from the internet. This field has undergone significant data paradigm evolution in recent years: from early reliance on synthetic data generated in simulation environments (such as AI2-THOR, Habitat), to mid-stage pretraining using human egocentric video, to the current pursuit of precisely aligned multi-modal data from real work scenarios. The APAC dataset represents the latest stage of this evolution.

As the robotics industry increasingly targets real positions in industrial and service sectors, training data must also make this "scene transition." While kitchen cooking is certainly a classic starting point for studying egocentric understanding, getting robots to actually work in garages, factories, and shipping centers requires high-quality, multi-modal, precisely aligned data collected in these environments. APAC's design philosophy aligns closely with the data requirements of large-scale robot foundation models like RT-2 (Google DeepMind's Robotics Transformer 2) and OpenVLA—these models need large amounts of real-world data covering diverse scenes with precise spatial information and semantic annotations to achieve cross-scene generalization.

Although 12 recordings with 248 clips is still limited in scale compared to larger datasets, its exploration of scene diversity, stereo vision, and multi-modal annotation provides a valuable paradigm reference for subsequent larger-scale industrial egocentric datasets. The standards it establishes in data specification (the combination of stereo vision + depth + tracking + captions) may become a blueprint for future large-scale industrial data collection. For teams working on robot manipulation learning, imitation learning, and embodied perception research, this is a data resource worth incorporating into experimental pipelines.

Key Takeaways