UniWALM: How a Unified World-Action-Language Model Breaks Through Robot Control Bottlenecks

Robot manipulation has long faced a core contradiction: language models excel at reasoning but lack physical intuition, while video-action models can execute skills but struggle with long-horizon planning. Simply stitching the two together often leads to a disconnect between reasoning and execution, with errors accumulating. UniWALM proposes a more radical approach—letting video, action, and language share representations within a single Transformer backbone, where each modality serves as both a conditioning signal and a supervision target.

Architectural Innovation: From Multi-Stage Stitching to Single-Backbone Collaboration

Previous methods typically use a vision-language model (VLM) as a planner that outputs text instructions, which are then handed to a video-action model for execution. This two-stage pipeline suffers from information loss: the VLM cannot see action execution details, and the action model cannot feed back uncertainty during execution. UniWALM's approach is to equip each modality with a diffusion head within a single mixture-of-transformers backbone, allowing the video, action, and language diffusion heads to denoise synchronously. This means the model decodes subtask descriptions and textual memory in the same forward pass that generates action trajectories, eliminating the need for an independent planning stage.

The direct benefit of this design is eliminating the interface bottleneck between stages. The generated language is re-consumed in the next step, forming a closed loop of "language as memory"—the model's own generated text becomes context for subsequent decisions, without relying on external planners or memory banks. This mimics how humans use language as working memory to guide action.

Mixture-of-transformers backbone: Refers to a large Transformer in which dedicated subnetworks (such as independent attention heads or feed-forward layers) are allocated to different modalities or tasks while sharing underlying representations. Similar to mixture-of-experts (MoE), it allows the model to maintain parameter efficiency while letting each modality focus on its own characteristics. UniWALM uses this to let the video, action, and language diffusion heads work in parallel while exchanging information through the shared backbone, avoiding the interface losses between fully separate models.

Flow Matching with Correction: Keeping the Action Flow Reliable Under Rich Objectives

Joint training of three modalities introduces a risk: the action generation flow may be "diluted" by language and video objectives, leading to degraded execution precision. UniWALM introduces the Flow Matching with Correction (FMC) training objective to address this. FMC teaches the model during training to recover from perturbed action trajectories, but adds no extra computational cost at inference time. In other words, it internalizes error-correction capability into the action flow itself, rather than adding a post-processing module at the inference stage.

This design philosophy is noteworthy: it does not avoid the conflicts of multi-task learning, but instead uses training objective design to keep the action flow robust under richer supervision. Based on the results reported in the paper, this strategy is effective.

Flow Matching is a class of generative modeling methods that learns a continuous-time velocity field to map a simple distribution (such as Gaussian noise) to a complex data distribution (such as action trajectories). Unlike diffusion models that denoise step by step, it directly regresses a conditional vector field, resulting in more stable training and more efficient sampling. UniWALM models action generation as a flow matching problem, enabling action trajectory generation to collaborate with video and language diffusion processes within a unified framework.

Perturbed trajectory recovery: FMC applies noise or perturbations to action trajectories during training and requires the model to predict the recovery direction. This is similar to data augmentation in denoising autoencoders or contrastive learning, but operates in continuous action space. Through this approach, the model learns to resist interference from language and video objectives during joint training, maintaining action flow precision without additional computation at inference.

Experimental Results: Competitiveness Without Pretrained Language Models

On the compositional pick-and-place task MolmoSpaces, UniWALM consistently outperforms same-backbone baselines (such as FastWAM). More noteworthy is its performance on the memory-dependent scenario RMBench: UniWALM achieves a 34.5% success rate without any pretrained language or reasoning model, surpassing Mem-0's keyframe-assisted setting (28.5%), which uses only single anchor frames. This means the representational capacity provided by the unified architecture itself can, to some extent, substitute for the prior knowledge of external language models.

Results on the real-world Find-the-Item task are even more convincing: generated textual memory raises the success rate from 11.1% to 61.1%, while state-of-the-art VLA models perform at or below random level (≤33.3%) on this task. This comparison shows that language generation is not merely an embellishment but a key mechanism for solving memory-dependent tasks.

Why "Language as Memory" Is More Effective Than External Planners

In traditional approaches, the language planner and action executor are separate; the planner cannot perceive execution errors, and the executor cannot request re-planning. In UniWALM, language generation occurs in the same propagation pass as action denoising, and the generated text directly encodes the current action context and is re-injected in the next step. This tight coupling makes language truly part of action generation rather than an external instruction.

From a broader perspective, UniWALM responds to a fundamental question in embodied intelligence: is it necessary to decouple reasoning from execution? At least in this work, the answer leans toward no—unified representation plus synchronous generation can achieve stronger memory and compositional capabilities without relying on external large models. Of course, this does not mean that scaling up models or incorporating external knowledge is worthless; rather, it points to another viable technical path.

Limitations and Outlook

The tasks UniWALM currently demonstrates are concentrated on pick-and-place and item finding; more complex long-horizon tasks and generalization in open environments still need verification. In addition, whether the training cost and stability of a single backbone with multiple diffusion heads still hold at larger scales is also a question worth monitoring. But based on existing evidence, it offers a compelling unified modeling paradigm: letting language, vision, and action supervise each other within the same representation space, rather than calling upon one another.

One-line Summary

UniWALM unifies video, action, and language in a single Transformer backbone, achieving a closed loop of language as memory.

Paragraph Summary

To address the disconnect between reasoning and execution in robot manipulation, UniWALM proposes unifying video, action, and language modalities within a single mixture-of-transformers backbone, each with a diffusion head for synchronous denoising, eliminating the independent planning stage. Generated language is re-consumed in the next step, forming a "language as memory" closed loop. To counter the dilution of action precision from joint training, it introduces the Flow Matching with Correction training objective, enabling the action flow to recover from perturbations with zero extra inference cost. Experiments show a 34.5% success rate on the RMBench memory task, surpassing Mem-0's 28.5%; on the real-world Find-the-Item task, generated textual memory raises the success rate from 11.1% to 61.1%, far exceeding advanced VLA models. This work demonstrates that unified representation and synchronous generation can substitute for external language model priors, offering a new path for embodied intelligence.