Dreamer

Institutions such as DeepMind and Meta AI have produced representative world model works including DreamerV3 and JEPA

In recent years, representative work on world models includes the Genie and Dreamer series and various video prediction foundation models.

Dreamer adopts the "KL balancing" technique, applying different gradient directions to the posterior and the prior respectively, so that the two move toward each other dynamically without collapsing.

In Dreamer reproduction practice, the stochastic component of the latent state requires appropriate KL constraints: too strong loses information, too weak leads to posterior collapse; the imagination trajectory length and batch size together determine gradient quality, and both need to be adjusted according to the task.

Dreamer's policy learning adopts the Actor-Critic framework: the Actor outputs an action distribution based on the imagined latent state s_t, and the Critic estimates the value of that state; gradients are backpropagated through the world model's latent trajectory, but the world model parameters are usually frozen to avoid the policy optimization signal contaminating the world model's learning objective, thereby maintaining training stability.

The cost of freezing the world model training strategy is: if the world model itself has bias, the policy will overfit to that bias, i.e., the phenomenon of 'model exploitation' occurs.

The Dreamer series adopts the approach of training policies within simulation, amortizing planning costs into the training phase.

The Dreamer series of work allows agents to learn policies entirely in latent space without rendering real pixels.

Representative early work on world models includes DeepMind's Dreamer series and the environment model used by OpenAI in Dota 2.

The Dreamer series (DreamerV1/V2/V3) pushed the world model framework toward more complex continuous control tasks by introducing discrete latent variables and improved world model loss functions.

The Dreamer series achieved performance comparable to or even better than model-free reinforcement learning on multiple Atari and 3D environment benchmarks.

A world model can be an explicit physics simulator (such as MuJoCo or Isaac Sim) or a learned implicit model (such as the recurrent state-space model used by the Dreamer series of algorithms).