What Can't 30,000 Hours of First-Person Video Teach a World Model?

World models are often seen as an alternative to physics simulators, but they are still a long way from practical deployment. A new paper uses 30,000 hours of first-person human video to directly test how far the "brute-force data scaling" path can actually go.

The research team built a dataset of rare scale: 30,000 hours of first-person video, covering over 1,000 scene types, filmed by 14,000 contributors. Rather than relying on opaque downstream metrics, they directly evaluated agent fidelity and object interaction fidelity on a difficult out-of-distribution benchmark.

A 100x Data Increase, But Asymmetric Results

Scaling training data by 100x improved both fidelity metrics, but the gains were extremely uneven.

Agent modeling performed quite well, while object fidelity was not only far lower than agent fidelity but also improved much more slowly. This means the model makes rapid progress in learning "how people move" but still struggles to understand "how objects change."

The authors further found that gains in agent capability may not come from the data itself. A carefully designed visual conditioning scheme can saturate agent fidelity using only a small fraction of the data. This allowed them to isolate object fidelity for separate measurement and find its own saturation point.

What Is Fidelity?

In this type of world model research, "fidelity" does not refer to image sharpness but rather the degree to which the model's generated future states align with real video along specific dimensions. The paper breaks it into two orthogonal dimensions: agent fidelity measures whether the model can accurately reproduce human poses, motion trajectories, and behavioral intent; object fidelity measures whether the model can correctly predict how objects change state under external forces or interactions—for example, whether a pushed cup topples along a plausible trajectory, or whether a grasped object follows the hand's motion.

The emphasis on "out-of-distribution benchmarks" is because evaluating within the training distribution tends to overestimate model capability: the model may simply have memorized statistical patterns of common scenes. Out-of-distribution benchmarks require the model to generalize to unseen scenes and interaction combinations, thus better exposing the gap between "looks right" and "physically right." This also explains why agent fidelity can saturate quickly—human motion patterns are relatively limited and highly repetitive, whereas the combinatorial space of object dynamics is far larger.

Supervised Schemes Shift Capacity, But the Gap Remains

Since data volume is not the only variable, the training approach becomes key.

The researchers proposed a new supervised scheme that shifts model capacity from "scene appearance" toward "object dynamics." This adjustment did improve object fidelity, but the authors also acknowledge that a significant gap still exists.

In other words, getting the model to focus less on "what the picture looks like" and more on "how things move" is the right direction, but it is not enough to close the chasm.

Visual Conditioning Schemes and Capacity Allocation

"Visual conditioning scheme" refers to how visual context is provided to the model when generating or predicting future frames—for example, which frames are used as conditioning inputs, how these conditions are encoded, and whether additional intermediate representations are introduced. The paper found that a carefully designed conditioning scheme can saturate agent fidelity with far less than 30,000 hours of data, indicating that a substantial portion of the previously observed gains actually came from conditioning design rather than data scale itself.

"Shifting model capacity from scene appearance toward object dynamics" is a reallocation of supervision signals. A model's total parameters and compute budget are limited. If the training objective primarily rewards reconstructing background textures, lighting, and static layout, capacity will settle on "scene appearance"; by adjusting the supervision objective so that gradients come more from object state prediction errors, the model will direct its representational capacity toward physical dynamics. This operation can be understood as injecting an inductive bias, but it cannot create information that does not exist in the data—if the diversity of object interactions in the video is insufficient, no amount of capacity shifting can break through the ceiling.

Conclusions Transfer to Humanoid Modeling, Pointing to the Same Problem

These conclusions are not limited to one type of model. After validation, the authors found that the same patterns transfer to downstream humanoid modeling tasks.

This leads to a more general judgment: scaled first-person data can push agent modeling close to its limit, but its impact on the world lags far behind.

The Relationship Between Humanoid Modeling and Out-of-Distribution Benchmarks

Humanoid modeling typically refers to having an agent control a humanoid body in simulation or the real world to complete locomotion and manipulation tasks. One of its core difficulties is predicting the coupled consequences between its own actions and environmental objects. The paper's conclusions transfer to this downstream task because both share the same bottleneck: modeling "how people move" can be rapidly improved with large amounts of repetitive human motion data, while modeling "how objects change" requires covering sufficiently diverse object properties, contact geometries, and mechanical outcomes—a type of sample whose distribution in naturally filmed first-person video is extremely imbalanced.

First-person perspective introduces additional challenges: the cameraperson typically focuses on the target ahead, causing the contact region between hands and objects to be frequently occluded or passed over quickly, making it difficult for the model to obtain clear interaction supervision signals. Therefore, even with a total data volume of 30,000 hours, the proportion of clips that actually contain high-quality object interactions may be far lower than the surface number suggests—one potential reason why object fidelity improves slowly.

How to Close the Gap? It's Not Just About Data Volume

The paper's core conclusion is quite measured: closing the gap between agent modeling and object modeling will depend on how the model is trained, not just on how much data it has seen.

This pours cold water on the current "data scale above all" narrative. 30,000 hours, over 1,000 scene types, 14,000 contributors—this data scale is already quite substantial, yet object fidelity still improves slowly. When the marginal returns of data rapidly diminish along a certain dimension, the importance of training objectives, capacity allocation, and conditioning design becomes prominent.

For researchers trying to train world models with video data, the value of this work may not lie in providing a specific solution but in clarifying the capability boundaries of "stacking data": what it can solve, and what it cannot.

One-line Summary

30,000 hours of first-person video shows that scaling data rapidly improves agent fidelity but struggles to improve object fidelity.

Paragraph Summary

The research team built a dataset of rare scale using 30,000 hours of first-person video covering over 1,000 scene types and filmed by 14,000 contributors, and directly evaluated agent fidelity and object fidelity on a difficult out-of-distribution benchmark. Results show that after a 100x data increase, both metrics improved but extremely unevenly: agent modeling performed well and improved rapidly, while object fidelity was low and improved slowly. Further experiments showed that gains in agent capability may not come from the data itself—a carefully designed visual conditioning scheme can saturate it with a small amount of data. The authors accordingly proposed a new supervised scheme that shifts model capacity from scene appearance toward object dynamics. Although it improved object fidelity, a significant gap remains. These patterns also transfer to downstream humanoid modeling. The paper's core conclusion is: closing the gap between agent and object modeling depends on the training approach, not merely on data volume.