DQN

DQN was proposed by DeepMind in 2013 and was the first to combine deep neural networks with Q-learning.

DQN是DeepMind在2013年提出的算法，首次将深度神经网络与Q-learning强化学习相结合，使AI能学会玩49款Atari游戏

Stable-Baselines3 is rebuilt on PyTorch and provides production-grade implementations of algorithms such as PPO, DQN, and SAC.

The core innovation of DQN lies in combining deep learning's perceptual capabilities with the Q-learning decision-making framework, and introducing two key techniques—experience replay and target networks—to stabilize the training process.

DeepMind published a paper in 2013 demonstrating a system capable of teaching itself to play Atari games.

Modern deep reinforcement learning methods (such as DQN, PPO, and SAC) can be viewed as function approximation generalizations of classical methods such as value iteration and policy iteration in continuous high-dimensional spaces.

DeepMind's DQN appeared on the cover of Nature in 2015, being the first to demonstrate that a single architecture could surpass human-level performance across multiple Atari games.

Pac-Man is one of the 49 Atari games tested by DQN, but DQN's performance on Pac-Man was relatively poor.

The DQN system was able to reach or surpass human-level performance on 49 Atari games using only raw pixel input.

DQN (Deep Q-Network) is a value-function-based method that uses a neural network to estimate the long-term value of each state-action pair.

The adaptation advantage of self-discovered rules over DQN may stem solely from differences in default configurations for data freshness management, rather than from internally evolved stronger plasticity mechanisms.

When the environment changes, controlling replay retention reverses the apparent adaptation advantage of self-discovered rules over DQN.

Replay Ratio (UTD) refers to the number of gradient update steps the Learner performs per step of environment data collected; standard DQN typically uses a value of 1 or lower.

When training a small DQN on a single environment on CPU, throughput is approximately 400 steps per second and cannot be improved by increasing the number of threads.

For off-policy algorithms such as DQN, the most reliable architecture is to have multiple Workers collect data only and write it into a shared replay pool, with the learner maintained as a single instance sampling from the buffer to perform updates.

The DQN-paradigm reinforcement learning approach requires millions of steps of self-play training, consumes enormous resources, and the trained results are difficult to transfer to new games.

Standardized environment interfaces allow researchers to directly connect off-the-shelf algorithms such as PPO, DQN, and A2C to the environment for training without needing to rewrite adapter code for each game

Model-free reinforcement learning methods (such as Q-Learning and PPO) learn policies or value functions directly from interactions with the environment without explicitly modeling the environment's dynamics.

DeepMind's DQN agent achieved superhuman performance in multiple Atari games, but minor visual perturbations can cause performance to collapse.

In the shaped reward experiment, a 97% single-cell occupancy rate was observed, with the agent remaining almost motionless on the cooldown tile.