Dreamer V1
A model-based reinforcement learning algorithm that learns a latent world model through a recurrent state-space model (RSSM) and trains policies by imagining trajectories in latent space, significantly improving sample efficiency.
Timeline (last 90 days)
Dreamer V1's world model consists of four parts: the representation model, the transition model, the reward prediction head, and the observation prediction head, all built on RSSM, corresponding respectively to posterior inference, prior prediction, and supervision signal generation.
Dreamer V1's loss function consists of a weighted sum of a reconstruction term, a KL divergence term, and a reward prediction term: the reconstruction term usually uses mean squared error or negative log-likelihood, and the reward prediction term uses cross-entropy or MSE.
In Dreamer V1, the KL divergence term is used to constrain the distance between the posterior distribution q(z_t|h_t,o_t) and the prior distribution p(z_t|h_t); if the KL weight is too large, the posterior is forced to approach the prior, causing the latent variables to lose observational information (information loss), and if the weight is too small, the transition model cannot generate meaningful imagined trajectories from the prior (posterior collapse).
There was recently a share on the Reddit community: the author wrote an illustrated tutorial on Dreamer V1 and open-sourced a matching PyTorch implementation, mapping the architecture, formulas and code item by item, with the article and code structure kept consistent.
The sample efficiency advantage of Dreamer V1 over model-free methods comes from the compressed reuse of observations; compared with earlier models based on pixel reconstruction, it makes predictions directly on the latent state, avoiding wasting capacity on irrelevant details.
All Facts (5)
Dreamer V1's world model consists of four parts: the representation model, the transition model, the reward prediction head, and the observation prediction head, all built on RSSM, corresponding respectively to posterior inference, prior prediction, and supervision signal generation.
50%UnverifiedDreamer V1's loss function consists of a weighted sum of a reconstruction term, a KL divergence term, and a reward prediction term: the reconstruction term usually uses mean squared error or negative log-likelihood, and the reward prediction term uses cross-entropy or MSE.
50%UnverifiedIn Dreamer V1, the KL divergence term is used to constrain the distance between the posterior distribution q(z_t|h_t,o_t) and the prior distribution p(z_t|h_t); if the KL weight is too large, the posterior is forced to approach the prior, causing the latent variables to lose observational information (information loss), and if the weight is too small, the transition model cannot generate meaningful imagined trajectories from the prior (posterior collapse).
50%UnverifiedThe sample efficiency advantage of Dreamer V1 over model-free methods comes from the compressed reuse of observations; compared with earlier models based on pixel reconstruction, it makes predictions directly on the latent state, avoiding wasting capacity on irrelevant details.
50%UnverifiedThere was recently a share on the Reddit community: the author wrote an illustrated tutorial on Dreamer V1 and open-sourced a matching PyTorch implementation, mapping the architecture, formulas and code item by item, with the article and code structure kept consistent.
50%