LaLa-World: Scaling Latent World Models with Natural Language Actions

World model research has long faced a dilemma: either fine-tune for a single domain, or rely on expensive large-scale video generation. LaLa-World offers a middle path—using language as a unified interface that lets a Latent World Model (LWM) learn across domains while achieving efficiency advantages in both planning and policy fine-tuning.

Core Idea: Translating Both Actions and Visual Changes into Language

The key design of LaLa-World is not complicated, but it is clever. When traditional JEPA-style latent world models predict the future, the action signal is typically a numerical vector, and the action spaces of different robots and different simulators are mutually incompatible, so data cannot be mixed. LaLa-World's approach is to describe both numerical actions and visual changes in natural language, so that data from simulation, real robot trajectories, and even ordinary images and videos can all be digested by the same future predictor.

This is essentially finding a "lingua franca" for heterogeneous data. Here, language is not just a conditioning input but a medium for data alignment. The authors name this model the Language-conditioned Latent World model, or LaLa-World for short.

JEPA (Joint Embedding Predictive Architecture) is a class of self-supervised learning architectures proposed by Yann LeCun. Its core idea is to make predictions in latent representation space rather than directly reconstructing pixels. Unlike generative models (such as video diffusion models), JEPA does not require the predictor to reproduce every pixel detail of the future; instead, it predicts the abstract representation of future frames in the encoder space. This design makes the model focus more on semantic-level changes, avoiding wasting capacity on unpredictable texture noise, thereby improving prediction efficiency and generalization. JEPA-style latent world models typically consist of an encoder, a predictor, and an action-conditioning module, with a low-dimensional latent vector as the prediction target. LaLa-World inherits exactly this line: its "future predictor" outputs not future images but future latent states, so both training and inference costs are significantly lower than those of pixel-level video generation models.

Second Design: Using a Language Model as the Future Predictor

Language descriptions alone are not enough; the predictor itself must be able to understand these descriptions. LaLa-World builds its future predictor on top of a language model, enabling it to parse language conditions when predicting future latent states. These two designs work in tandem: language unifies the data format, and the language model provides semantic understanding.

According to the paper, this combination allows a single predictor to be trained jointly on multi-domain, multi-task data, rather than training a separate model for each domain as existing LWMs do.

Planning Performance: Cross-Domain Average Outperforms Specialized Models

On MPC (Model Predictive Control) planning tasks, LaLa-World's average performance exceeds that of existing latent world models trained separately for each domain or task. In more detail, adding image and video data further improves average planning performance while also making the model more robust to action phrasing—in other words, even when the same action is described in different ways, the model's performance is more stable.

The paper also provides a matched comparison with the RAEv2 video generation model baseline: LaLa-World achieves comparable planning performance with only 1/6 of the training cost and 13.9x faster planning. Here, "matched comparison" means the two are compared under equivalent conditions, so the efficiency gap is meaningful.

MPC (Model Predictive Control) is a model-based closed-loop control method: in each control cycle, the controller uses the current state and the world model to predict the evolution over several future steps and solves an optimization problem over a finite horizon to obtain an optimal action sequence; however, it executes only the first action in that sequence, then re-observes, re-predicts, and re-optimizes in the next cycle. This receding-horizon strategy gives the system a degree of robustness to model errors and external disturbances. In world-model-based planning, the quality of MPC depends heavily on the accuracy of the future predictor—the more accurate the prediction, the more reliable the optimized action sequence. LaLa-World evaluates average performance on MPC planning tasks, which essentially tests the fidelity of its latent world model as a "simulator" and whether language conditioning enables a single model to have usable predictive capability across multiple domains.

From Planning to Policy: Transfer Performance of LaLa-Policy

The pretrained LaLa-World can also serve as initialization for LaLa-Policy—a policy model that jointly predicts actions and future visual representations. On the two benchmarks LIBERO and RoboCasa, LaLa-Policy performs comparably to Cosmos Policy while being up to 5.5x faster at inference.

This shows that a language-conditioned latent world model can not only perform planning but also serve as an effective starting point for downstream policy learning. For robotic application scenarios that require fast inference, this speed advantage has practical significance.

LIBERO is a commonly used lifelong learning benchmark in the field of robotic manipulation, containing multiple task suites (such as Spatial, Object, Goal, and Long) for evaluating a policy's learning and transfer ability across task sequences; RoboCasa is a large-scale robotic manipulation simulation benchmark for everyday household scenarios, emphasizing multi-task, multi-object, and long-horizon manipulation. These two benchmarks are often used to measure the generalization and sample efficiency of policy models. Cosmos Policy is a policy baseline built on NVIDIA's Cosmos world foundation model, which leverages large-scale video pretraining to acquire physical common sense and is then fine-tuned into a robot policy. LaLa-Policy performs comparably to Cosmos Policy on the same benchmarks but is 5.5x faster at inference, indicating that a language-conditioned latent world model, when used as policy initialization, can substantially reduce inference overhead while maintaining performance—which is especially critical for real-time robot control.

A Few Observations

The value of LaLa-World lies in providing a "scalable" paradigm: language as a common representation of actions and vision frees the world model from being locked into a single data source. However, the quality and coverage of language descriptions directly affect model capability—the paper mentions improved robustness to action phrasing, but also implies the sensitivity of the language interface itself. In addition, the experiments are concentrated on planning benchmarks and robotic manipulation tasks, and generalization in more open scenarios remains to be verified.

Overall, this work offers a pragmatic and efficient direction for cross-domain scaling of latent world models, and the numbers on training cost and inference speed are convincing enough.

One-line Summary

LaLa-World uses language to unify heterogeneous action and visual data, enabling a single latent world model to learn across domains and plan efficiently.

Paragraph Summary

LaLa-World proposes using natural language as a common representation of actions and visual changes, enabling heterogeneous data from simulation, real robots, and ordinary videos to be jointly trained by a single language-conditioned latent world model. Its future predictor is built on a language model, capable of parsing language conditions and predicting future latent states. On MPC planning tasks, its average performance exceeds that of LWMs trained separately per domain, and robustness improves further when image and video data are added; compared with the RAEv2 video generation baseline, training cost is 6x lower and planning is 13.9x faster. The pretrained model can also initialize LaLa-Policy, which performs comparably to Cosmos Policy on LIBERO and RoboCasa while being up to 5.5x faster at inference.