Thinker-WM
A unified world model that encodes RGB vision, metric depth, and robotic actions into a shared latent dynamics representation through a Unified Dynamics Transformer. It supports a range of capabilities, including joint video-action generation, world prediction, policy reasoning, action-conditioned generation, video-to-action generation, and inverse dynamics.
Timeline (last 90 days)
Thinker-WM uses a shared Unified Dynamics Transformer to unify RGB, metric depth, and robot actions into the same dynamic representation, while equipping each modality with dedicated encoders and decoders to preserve the output structure of each modality.
The same set of parameters in Thinker-WM can support multiple functions, including joint video-action generation, world prediction, policy reasoning, action-conditioned generation, video-to-action generation, and inverse dynamics.
Action probe experiments show that even when Thinker-WM's visual representations are frozen, action information can still be recovered from them, indicating that the model preserves motion-related structure at the internal representation level.
Thinker-WM's training corpus totals approximately 20K hours of first-person video, 10K hours of real robot data, and 1K hours of simulation data, of which about 4K hours of real robot data has aligned depth supervision.
Thinker-WM ranks first on the embodied world model evaluation benchmarks EWMBench, WorldOlympiad, and WorldArena.
Ablation experiments on Thinker-WM show that action conditioning, geometric interaction, and multi-function joint training all contribute positively to the shared representation.
The direction Thinker-WM points to is: world models for embodied intelligence do not need to split vision, geometry, and action into independent modules; a shared dynamics representation may be a more natural choice.
The challenges facing unified world models include the enormous differences in data distribution, temporal scale, and supervision density across modalities, and how to maintain the output quality of each modality under shared parameters remains a question that future work needs to answer.
All Facts (8)
Thinker-WM uses a shared Unified Dynamics Transformer to unify RGB, metric depth, and robot actions into the same dynamic representation, while equipping each modality with dedicated encoders and decoders to preserve the output structure of each modality.
50%UnverifiedThe same set of parameters in Thinker-WM can support multiple functions, including joint video-action generation, world prediction, policy reasoning, action-conditioned generation, video-to-action generation, and inverse dynamics.
50%UnverifiedAction probe experiments show that even when Thinker-WM's visual representations are frozen, action information can still be recovered from them, indicating that the model preserves motion-related structure at the internal representation level.
50%UnverifiedThinker-WM's training corpus totals approximately 20K hours of first-person video, 10K hours of real robot data, and 1K hours of simulation data, of which about 4K hours of real robot data has aligned depth supervision.
50%UnverifiedThinker-WM ranks first on the embodied world model evaluation benchmarks EWMBench, WorldOlympiad, and WorldArena.
50%UnverifiedAblation experiments on Thinker-WM show that action conditioning, geometric interaction, and multi-function joint training all contribute positively to the shared representation.
50%UnverifiedThe direction Thinker-WM points to is: world models for embodied intelligence do not need to split vision, geometry, and action into independent modules; a shared dynamics representation may be a more natural choice.
50%UnverifiedThe challenges facing unified world models include the enormous differences in data distribution, temporal scale, and supervision density across modalities, and how to maintain the output quality of each modality under shared parameters remains a question that future work needs to answer.
50%