VLM Agents Self-Evolve in Latent Worlds: A Deep Dive into the Dwell Framework

Vision-language models (VLMs) have already internalized rich world knowledge, but turning that knowledge into policy improvement for agents through online interaction has long been a challenge. The recently proposed Dwell framework offers an elegant answer: let the VLM predict the future and adjust its behavior within its own latent space, enabling efficient and continuous learning.

Why VLM Agents Need to "Learn in a Latent World"

Traditional model-based reinforcement learning (MBRL) typically relies on an explicit world model to predict future states, such as reconstructing the next frame. But for VLM agents, this approach has two bottlenecks: first, policy updates shift the interaction distribution, so the world model must adapt accordingly; second, continuously updating a large pretrained model is computationally expensive. Dwell's core insight is: a VLM doesn't need to reconstruct the future—it only needs to predict the future that is useful for action.

Model-based reinforcement learning (MBRL) is a class of methods in which an agent first learns a model of environment dynamics, then uses that model for planning or for generating simulated experience to improve its policy. Classic implementations (such as Dreamer and MuZero) typically require explicitly predicting the next observation or reward, and can handle long-horizon credit assignment. Their advantage is sample efficiency—the agent can trial-and-error in "imagination" without frequently interacting with the real environment; the cost is the coupling between world model and policy: policy updates cause data distribution shift, and the world model must continuously adapt, otherwise prediction errors accumulate and mislead policy optimization. Dwell targets exactly this coupling bottleneck by proposing to replace a separate world model with the VLM's own representations.

How Dwell Works: Turning the VLM into a Recurrent Latent Transition Operator

Dwell reframes the VLM as a recurrent latent transition operator, enabling it to predict, adjust, and act in latent space. Specifically, it leverages the VLM's own representational capacity so that the model plays both the role of world model and policy. The agent learns from real interactions, predicting and evaluating the outcomes of actions; at the same time, it learns from latent imagination to improve its own behavior. This dual-role design avoids the parameter and computational burden of an additional world model.

Combining Latent Imagination with Real Interaction

The key mechanism of Dwell is to let the agent "imagine" in latent space. Unlike pixel-level reconstruction, latent imagination focuses only on abstract features useful for decision-making, so computational efficiency improves markedly. The paper reports that Dwell achieves the strongest overall performance among trainable methods across five interactive visual environments, while greatly reducing the computation required for imagination. This validates an important point: an effective world model does not need to reconstruct the future—it should predict the future that is useful for action.

The recurrent latent transition operator draws on the recursive structure of hidden states over time in recurrent neural networks (RNNs): the agent maintains a latent state (z_t), and at each step, based on the current observation and action, uses the VLM to predict the next latent state (z_{t+1}) and the corresponding action value. Unlike pixel-level world models, this operator does not require the latent state to be decodable back into an image; it only requires that it retain information useful for decision-making (such as object positions and interactability). In this way, the same set of VLM parameters both encodes observations and performs transition prediction and value estimation, effectively compressing the world model and policy into a shared representation and thereby avoiding the parameter bloat and training instability brought by extra networks.

Latent imagination means the agent does not generate real images but instead rolls out predictions of several future states in latent space, and uses them to evaluate action values and update the policy. It complements real interaction: real interaction provides calibration signals that prevent latent predictions from drifting; latent imagination expands training data at very low computational cost, making it especially suitable for online continual learning. The substantial reduction in computation comes from two aspects: first, the decoder overhead of pixel-level reconstruction is eliminated; second, the dimensionality of latent space is far lower than that of images, so the matrix operations for rollout prediction are significantly reduced. This also explains why Dwell can maintain policy improvement without frequently updating a large VLM.

Experimental Performance and Open-Source Contribution

Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods and significantly reduces the computational demand of the imagination stage. This means VLM agents can achieve continuous policy improvement through latent-space rollouts without frequently updating large models. The research team has released the code on GitHub (https://github.com/Tom111-lhx/Dwell), providing a reproducible foundation for follow-up research.

Implications for Continual Learning in Agents

The significance of Dwell lies not only in an efficient RL framework, but also in demonstrating how to leverage the internal knowledge of a pretrained VLM to drive policy improvement. By unifying the world model and policy within the same VLM, the agent can cycle between real interaction and latent imagination to achieve continual learning. This approach offers new ideas for building next-generation agents capable of continuously expanding their abilities, and reveals the potential of latent-space prediction in interactive tasks such as robotics and autonomous driving.

As VLM capabilities grow, how to let them evolve autonomously in dynamic environments remains an open problem. Dwell offers a promising direction: let agents learn in their own latent worlds rather than reconstructing the pixel world.

One-line Summary

Dwell lets VLMs predict the future and adjust policies in latent space, enabling efficient continual learning.

Paragraph Summary

The Dwell framework reframes vision-language models (VLMs) as recurrent latent transition operators, making the VLM serve simultaneously as world model and policy, imagining and deciding in latent space rather than reconstructing pixel-level futures. This approach avoids the parameter and computational overhead of an additional world model in traditional model-based reinforcement learning, and addresses the problem of interaction distribution drift caused by policy updates. Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods while greatly reducing the computation required for the imagination stage. Its core insight is: an effective world model does not need to reconstruct the future—it should predict the future that is useful for action. The code has been open-sourced, providing a reproducible foundation for continual learning in VLM agents.