Dwell
A reinforcement learning framework for VLM agents that treats the VLM itself as a recurrent latent transition operator, predicting the future in latent space and improving the policy without requiring a separate world model.
Timeline (last 90 days)
The Dwell framework recasts the VLM as a recurrent latent transition operator, enabling the model to play both the role of world model and policy.
Dwell achieves the strongest overall performance among trainable methods across five interactive visual environments.
Dwell greatly reduces the computation required for imagination compared with pixel-level reconstruction methods.
Dwell's core insight is that the VLM does not need to reconstruct the future, only to predict the future that is useful for action.
Latent imagination in Dwell focuses only on abstract features useful for decision-making and does not require the latent state to be decodable back to images.
Dwell cycles between real interaction and latent imagination to achieve continual learning.
All Facts (6)
The Dwell framework recasts the VLM as a recurrent latent transition operator, enabling the model to play both the role of world model and policy.
50%UnverifiedDwell achieves the strongest overall performance among trainable methods across five interactive visual environments.
50%UnverifiedDwell greatly reduces the computation required for imagination compared with pixel-level reconstruction methods.
50%UnverifiedDwell's core insight is that the VLM does not need to reconstruct the future, only to predict the future that is useful for action.
50%UnverifiedLatent imagination in Dwell focuses only on abstract features useful for decision-making and does not require the latent state to be decodable back to images.
50%UnverifiedDwell cycles between real interaction and latent imagination to achieve continual learning.
50%