EmbodiedWorld-200K: Training Embodied Agents with a Video World Model

Embodied agents that want to move freely in the open world cannot rely on task-specific training in a single scene alone. The recently released EmbodiedWorld-200K dataset and its accompanying video world model training scheme offer a low-cost, scalable new path toward building general Embodied World Agents (EWA).

What Is an Embodied World Agent (EWA)?

The research team defines this class of agents as systems capable of perceiving, reasoning, and executing general actions across diverse 3D visual environments. Unlike most current work, which is confined to specific tasks and single scenes, the core capability of EWA is to understand visual input from the open world and complete navigation and planning in unseen environments.

The paper notes that existing research has not sufficiently explored general embodied agents in open-world scenarios, and that models generally transfer poorly to new environments. This is the starting point of the work: first solve the problem of open-world navigation and planning—being able to "understand what it sees and find its way through."

Embodied Agent refers to an intelligent system that has a physical or virtual body and can interact with the environment through a perception–decision–action loop, as distinguished from models that passively reason over static text or images. Its core difficulties lie in partial observability, continuous action spaces, and long-horizon planning. Open World further requires the agent to remain robust in scenarios outside the training distribution, which places higher demands on the generalization of visual representations and on spatial reasoning. Out-of-Distribution (OOD) generalization is a common metric for measuring such capability: how a model performs on unseen scenes, new layouts, or new instruction combinations. The navigation and planning emphasized in this article are precisely the most fundamental and most challenging part of open-world embodied tasks.

EmbodiedWorld-200K: A 200K-Scale Dataset with Multi-Level Annotations

To support offline pretraining and fair evaluation of EWA, the team built EmbodiedWorld-200K. The dataset contains more than 200,000 samples spanning a variety of scenes, and each sample comes with multi-level annotations ranging from low-level to high-level:

• Low-level annotations: camera motion trajectories, describing the agent's movement path through space;
• High-level annotations: semantic instructions, defining task goals and behavioral intent;
• Unified evaluation splits: standardized data splits and task-specific metrics to enable fair comparison across methods.

This two-layer structure of "trajectory + instruction" lets the model learn both spatial motion patterns and language-level task semantics, laying the foundation for open-world navigation and planning.

The Video World Model as a Low-Cost Interactive Environment

Offline data can only provide supervision signals, whereas the essential characteristic of EWA is interaction with the environment. With offline training alone, models struggle to generalize to scenarios outside the training distribution. But building dedicated real interactive environments is extremely expensive, and this has become the main bottleneck constraining interactive training.

The team's proposed solution is to use a video world model as a low-cost proxy for the environment, and perform interactive reinforcement learning (RL) post-training of EWA within it. The agent executes actions, receives feedback, and optimizes its policy inside the "virtual environment" generated by the world model, thereby bypassing the expense of building real environments.

The value of this approach lies in turning the "environment" into a learnable, reusable model component—the quality of the world model directly determines the effectiveness of post-training, while also opening the possibility of extending to more scenes in the future.

Video World Model is a class of prediction models centered on video generation, whose goal is to learn the state-transition rules of an environment: given the current observation and the agent's action, predict the visual observation at the next moment. Unlike dynamics models that only predict single frames, a video world model needs to maintain temporal consistency, spatial geometric plausibility, and long-horizon coherence. In embodied intelligence, it is regarded as a differentiable, queryable "learned simulator"—the agent can trial and error at low cost within it, without needing a real robot or a simulation engine. Its typical training method is self-supervised prediction on large-scale video data, and at inference time it generates future frames segment by segment in an autoregressive or diffusion manner. The key challenges of this paradigm lie in action-conditioned modeling, error accumulation, and evaluation of generation quality, while generation quality and action-alignment accuracy directly determine whether RL post-training can yield a reliable policy.

Experimental Results: Dual Benefits of Pretraining and RL Post-Training

Empirical results yield conclusions at two levels:

1. Pretraining on EmbodiedWorld-200K significantly improves EWA's open-world embodied planning capability;
2. Further RL post-training inside the world model brings clear gains on out-of-distribution scenarios.

The second point is especially critical—it shows that interactive post-training can indeed compensate for the generalization shortcomings of offline data, validating the effectiveness of the "world model as environment proxy" training paradigm.

Open-Source Resources and Future Impact

The team has released the dataset, baseline model checkpoints, and training and evaluation code, with the project homepage at xiaokunfeng.github.io/EmbodiedWorld-200K.

From the perspective of research trends, using video generation models as training environments for embodied intelligence is becoming an important direction for reducing the cost of interaction data. The contribution of EmbodiedWorld-200K is not just data scale, but that it links "dataset pretraining + RL post-training within a world model" into a complete training pipeline, providing a reproducible baseline for research on open-world embodied agents.

For researchers and developers interested in the intersection of embodied intelligence, world models, and reinforcement learning, this open-source resource is worth following.

One-line Summary

EmbodiedWorld-200K advances open-world embodied agents with 200K-scale annotated data and RL post-training within a video world model.

Paragraph Summary

This article introduces the EmbodiedWorld-200K dataset and its accompanying video world model training scheme, aimed at building general Embodied World Agents (EWA). The dataset contains over 200,000 samples and provides multi-level annotations ranging from low-level to high-level, including camera motion trajectories and semantic instructions, along with unified evaluation splits. The team proposes using a video world model as a low-cost environment proxy, in which agents undergo interactive reinforcement learning post-training, bypassing the expense of building real environments. Experiments show that pretraining on this dataset significantly improves open-world embodied planning capability, while RL post-training within the world model brings clear additional gains on out-of-distribution scenarios, validating the effectiveness of the "dataset pretraining + RL post-training within a world model" pipeline. The dataset, baseline checkpoints, and training and evaluation code are all open-sourced.