DreamX: Turning Video Diffusion Models into General-Purpose Latent Action Learners

Video diffusion models are typically used to generate realistic imagery, but a new study is trying to make them learn to understand actions. DreamX reuses the spatiotemporal features of pretrained video diffusion models to extract compact latent action representations, building a shared action interface that spans embodiments and environments.

The Core Challenge of Action Representation

Learning general-purpose action representations from video requires a model to capture dynamics that remain consistent across different ways of executing the same action, while ignoring variations in embodiment morphology and appearance. Existing latent action models can infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual changes, limiting cross-embodiment transfer capability.

DreamX's starting point is precisely to solve this entanglement problem. Instead of relying on additional action annotations, it mines action-related dynamic information directly from within video diffusion models.

Core Techniques: 300x Compression and Generative Priors

DreamX's key design lies in two aspects. First, it extracts compact latent actions from the rich spatiotemporal features of video models, compressing each latent video frame by 300x (from 1,200 tokens down to 4 tokens). This extreme compression forces the model to retain only the information most critical to action while filtering out appearance details.

Second, leveraging the generative priors of the video model itself, these latent actions can capture action-related dynamics and decouple them from visual appearance. In other words, what the model learns is not "what the picture looks like" but "how the action changes the picture."

The direct result of this decoupling is that the resulting latent space can serve as a shared action interface across embodiments (humans and robots), across environments, and across action parameterizations.

Performance and Transfer Capability

On the LARY-Bench benchmark, DreamX achieves state-of-the-art performance among latent action models. Even more noteworthy is its transfer capability: an action encoded in one setting remains effective in another.

This latent space supports arbitrary-to-arbitrary video transfer as well as zero-shot action transfer across multiple embodiments and environments. This means that action representations learned from human videos can be directly used for robot policies without retraining an action encoder for the robot embodiment.

Moreover, this shared interface also enables policy learning in a compact latent action space, thereby reducing the computational cost of policies based on video diffusion models.

LARY-Bench is a benchmark for evaluating the quality of latent action representations. Its core idea is to examine the usability of learned action latent variables in downstream tasks, such as action recognition, cross-video action alignment, and conditional generation based on latent actions. It typically uses "whether actions are correctly encoded and can be reused across samples" as its evaluation dimensions, rather than simply looking at the similarity of reconstructed pixels. Therefore, achieving state-of-the-art performance on this benchmark means that DreamX's latent space not only has a high compression rate but also preserves semantically transferable action information.

Zero-shot action transfer refers to the model directly applying action representations learned from a source domain (e.g., human videos) to execution in a target domain (e.g., robots) without having seen any demonstration data from the target embodiment or target environment. Its feasibility depends on whether the action representation is decoupled from embodiment morphology—if the representation incorporates embodiment-specific information such as arm length or joint configuration, transfer will fail. DreamX achieves this through a shared latent action interface, essentially using an "action semantic space" independent of the executor as an intermediate language.

Significance for Embodied Intelligence

DreamX's approach reveals a trend: the value of video generation models may extend beyond generating content itself. Their internal features already implicitly encode an understanding of physical dynamics, and the key lies in how to distill these features into reusable action representations with the right constraints (such as extreme compression and generative priors).

Zero-shot action transfer across embodiments has long been a difficult problem in the field of embodied intelligence. The shared latent action space provided by DreamX offers a lightweight path toward unifying human demonstrations and robot execution, and lays the foundation for subsequent efficient policy learning in a compact space.

Background

Video Diffusion Models are a class of generative models that generate video by iteratively denoising across spatiotemporal dimensions, with a typical example being the architectural approach behind Sora. Compared with image diffusion models, they additionally model temporal dependencies between frames, so their internal features naturally encode information about how objects move, deform, and interact over time. Such models are typically pretrained on large amounts of internet video, and their feature spaces contain rich priors of physical dynamics. DreamX's insight is precisely this: rather than training an action encoder from scratch, treat this generative model that already implicitly understands dynamics as a feature extractor and "read" action representations from it.

Latent Action Models refer to models that do not rely on explicit action labels but instead infer actions from observed visual transitions (changes between consecutive frames). A common approach is to use an encoder to compress inter-frame differences into a low-dimensional vector, then require a decoder to reconstruct future frames from it. The difficulty is that visual transitions simultaneously contain action and appearance changes (lighting, texture, viewpoint); without constraints, the latent variable will mix the two together, causing the representation to fail after switching to another embodiment or environment. DreamX applies constraints through extreme compression and generative priors precisely to alleviate this entanglement.

One-line Summary

DreamX reuses the spatiotemporal features of video diffusion models to extract 300x-compressed cross-embodiment latent action representations.

Paragraph Summary

DreamX proposes a method for extracting general-purpose action representations from pretrained video diffusion models. To address the problem that existing latent action models entangle actions with visual appearance changes and have limited cross-embodiment transfer, DreamX extracts compact latent actions from the spatiotemporal features of video models, compressing each latent video frame from 1,200 tokens to 4 tokens (300x), forcing the model to retain only action-critical information, and uses generative priors to decouple action from appearance. The resulting latent space can serve as a shared action interface across embodiments, environments, and action parameterizations. On LARY-Bench it achieves state-of-the-art performance among latent action models, supports arbitrary-to-arbitrary video transfer and zero-shot action transfer, and also reduces the computational cost of policy learning based on video diffusion models, providing a lightweight path toward unifying human demonstrations and robot execution.