P³-WAM: A Unified World-Action Model for Egocentric Dexterous Manipulation

Egocentric dexterous manipulation has long been fragmented: perception, prediction, and planning each operate in isolation, with models often specializing in only one of them. A new study named P³-WAM attempts to break down these barriers with a unified architecture, allowing all three capabilities to reinforce one another under a single set of weights.

One Model, Three Modes

The core design of P³-WAM is three complementary modes — "Perceive, Predict, Plan" — that share a single video diffusion Transformer backbone. According to the paper, the Perceive mode reconstructs articulated hand motion and camera trajectories from video; the Predict mode generates future visual frames conditioned on given actions; and the Plan mode, conditioned on an initial image, language instructions, and optionally a visual goal, simultaneously generates action trajectories and their expected visual consequences.

The ingenuity of this design lies in the fact that all three tasks share the underlying representation while preserving their individual characteristics through modality-specific adaptation layers. The model injects a compact structured 3D action representation into a pretrained video diffusion Transformer, adopting a mixture-of-transformers architecture — shared base weights, modality-specific adaptation, and bidirectional vision-action attention.

Video Diffusion Transformer is a mainstream architecture in the video generation field in recent years, combining the step-by-step denoising process of diffusion models with the attention mechanism of Transformers. Traditional diffusion models mostly rely on U-Net convolutional structures, while the Transformer variant directly models relationships between spatiotemporal tokens through self-attention, enabling more flexible handling of long sequences and cross-modal conditioning. In P³-WAM, the pretrained video diffusion Transformer provides strong visual priors, and the authors transform it into a versatile model handling perception, prediction, and planning simultaneously by injecting 3D action representations and adding modality-specific adaptation layers. This "pretrained large model + lightweight adaptation" paradigm is becoming a common path in the embodied intelligence field for reducing data requirements and improving generalization.

Mixture-of-Transformers (MoT) is a Transformer architecture variant inspired by Mixture of Experts (MoE). Unlike MoE, which sparsely activates experts in FFN layers, MoT typically refers to configuring independent attention or normalization adaptation modules for different modalities or tasks on top of shared backbone weights, thereby maintaining parameter efficiency while avoiding negative transfer between modalities. P³-WAM adopts this design, allowing visual and action tokens to share the underlying representation while preserving their individual characteristics through modality-specific adaptation and bidirectional vision-action attention, ensuring that the three modes do not interfere with each other during unified training.

Stochastic Modality Visibility: Making Video and Action Mutually Condition Each Other

The key mechanism of unified training is "stochastic modality visibility." Video and 3D action can serve as either conditioning context or prediction target across the three modes. This means the model cannot take the easy route of learning only one mapping, but must flexibly switch between different task configurations.

This training strategy yields direct benefits: experiments show that unified training outperforms single-mode training across all three capabilities, including improved planning quality under vision-language model evaluation. In other words, learning to predict future frames helps with better action planning, and learning to reconstruct hand motion helps understand how actions affect visual outcomes.

Stochastic Modality Visibility is a multimodal training strategy: in each forward pass, it randomly determines whether modalities such as video, action, or language serve as input conditions or prediction targets. This forces the model to learn bidirectional conditional distributions between modalities rather than fitting only unidirectional mappings. Similar ideas have been reflected in multimodal large models, such as randomly masking image or text tokens. In P³-WAM, this mechanism allows video and 3D action to flexibly swap roles across the three modes, thereby forming a mutual distillation effect among perception, prediction, and planning tasks — one of the key reasons why unified training outperforms single-mode training.

Cross-Dataset Performance: Surpassing Specialized Baselines

Across four datasets, P³-WAM surpasses specialized baseline models on both 3D hand reconstruction and action-conditioned video prediction tasks. In terms of planning, the goal-oriented manipulation videos it generates outperform video generation baselines in instruction following and action progression.

These results point to a conclusion: perception, prediction, and planning are not independent skills but unified capabilities that can mutually distill from one another. For robot learning and embodied intelligence, this unified modeling paradigm may be more scalable than stacking multiple specialized modules.

The paper's project page is publicly available: https://p3-wam.pages.dev/ . Code and model details are worth following for future updates.

Articulated Hand Motion Reconstruction refers to recovering the 3D rotation and translation of hand joints from video, typically involving more than 21 joint points. Unlike sparse representations that only estimate wrist position, articulated reconstruction outputs a complete skeletal pose that can directly drive robot dexterous hands or be used for motion understanding. P³-WAM's Perceive mode requires simultaneous reconstruction of hand joint motion and camera trajectory, meaning the model must decouple hand self-motion from camera motion, placing high demands on spatiotemporal understanding. This task is often used as a benchmark for egocentric manipulation perception to evaluate a model's ability to model fine-grained interactions.

One-line Summary

P³-WAM uses a unified video diffusion Transformer to integrate perception, prediction, and planning, surpassing specialized baselines on multiple tasks.

Paragraph Summary

P³-WAM proposes a unified architecture that integrates three capabilities in egocentric dexterous manipulation — perception, prediction, and planning — into a shared video diffusion Transformer backbone. Through three modes — Perceive, Predict, and Plan — it respectively reconstructs hand motion and camera trajectories, generates action-conditioned future frames, and produces action trajectories along with their visual consequences. Its core mechanism, "stochastic modality visibility," allows video and 3D action to serve as mutual conditions and targets under different task configurations, forcing the model to learn bidirectional mappings. Experiments show that unified training outperforms single-mode training across all three capabilities, and surpasses specialized baselines on 3D hand reconstruction and action-conditioned video prediction tasks across four datasets, with planning videos outperforming video generation baselines in instruction following and action progression.