LaLa-World
A language-conditioned latent world model that translates actions and visual changes into natural language, enabling joint training across domains and prediction of future latent states; it can be used for MPC planning and for initializing policy learning.
Timeline (last 90 days)
The full name of LaLa-World is Language-conditioned Latent World model.
LaLa-World describes both numerical actions and visual changes in natural language, enabling data from simulation, real robot trajectories, ordinary images, and videos to be digested by the same future predictor.
The future predictor of LaLa-World outputs not future images but future latent states, so training and inference costs are significantly lower than those of pixel-level video generation models.
LaLa-World builds its future predictor on top of a language model, enabling it to parse language conditions to predict future latent states.
A single predictor in LaLa-World can be jointly trained on multi-domain, multi-task data, without needing to train a separate model for each domain as existing LWMs do.
On MPC planning tasks, LaLa-World's average performance surpasses that of existing latent world models trained separately for each domain or task.
After adding image and video data, LaLa-World's average planning performance improves further, while its robustness to action phrasing becomes stronger.
Compared with the RAEv2 video generation model baseline, LaLa-World achieves comparable planning performance with only 1/6 of the training cost and 13.9 times faster planning.
A pretrained LaLa-World can serve as initialization for LaLa-Policy, a policy model that jointly predicts actions and future visual representations.
All Facts (9)
The full name of LaLa-World is Language-conditioned Latent World model.
50%UnverifiedLaLa-World describes both numerical actions and visual changes in natural language, enabling data from simulation, real robot trajectories, ordinary images, and videos to be digested by the same future predictor.
50%UnverifiedThe future predictor of LaLa-World outputs not future images but future latent states, so training and inference costs are significantly lower than those of pixel-level video generation models.
50%UnverifiedLaLa-World builds its future predictor on top of a language model, enabling it to parse language conditions to predict future latent states.
50%UnverifiedA single predictor in LaLa-World can be jointly trained on multi-domain, multi-task data, without needing to train a separate model for each domain as existing LWMs do.
50%UnverifiedOn MPC planning tasks, LaLa-World's average performance surpasses that of existing latent world models trained separately for each domain or task.
50%UnverifiedAfter adding image and video data, LaLa-World's average planning performance improves further, while its robustness to action phrasing becomes stronger.
50%UnverifiedCompared with the RAEv2 video generation model baseline, LaLa-World achieves comparable planning performance with only 1/6 of the training cost and 13.9 times faster planning.
50%UnverifiedA pretrained LaLa-World can serve as initialization for LaLa-Policy, a policy model that jointly predicts actions and future visual representations.
50%