DQN for Beginners: Concepts, Code, and Math Explained

Deep reinforcement learning (RL) often intimidates newcomers, especially when DQN (Deep Q-Network) enters the picture—the triple entanglement of concepts, code, and math can be bewildering. Recently, an author on Reddit posted a one-page DQN note attempting to break down this classic algorithm in the simplest way possible. Based on that note, this article combines DQN's core ideas and implementation details to lay out a clear learning path for you.

DQN Core Concepts: From Tables to Functions

Traditional Q Learning relies on a table to store the Q-value for each state-action pair, but when the state space is continuous or enormous, the tabular approach becomes infeasible. DQN's key innovation is to approximate the Q-function with a neural network—learning a parameterized function that takes a state (or state-action pair) as input and outputs the corresponding Q-value. This way, there's no need to explicitly store all Q-values; the network can generalize to unseen states. The author emphasizes in the note that DQN essentially transforms Q Learning's update target into a regression problem, training the network by minimizing the gap between predicted Q-values and target Q-values.

This shift brings two core challenges: correlation between samples and non-stationarity of target Q-values. To address these, DQN introduces two major mechanisms: Experience Replay and Target Network. Experience Replay breaks data correlation by storing past transitions and randomly sampling them, while the Target Network periodically copies the main network's parameters to stabilize the training target. These concepts are concisely explained in the note.

Relationship between Q Learning and DQN: Q Learning is a model-free RL algorithm that learns the value of each state-action pair by maintaining a Q-table. Its update rule is based on the Bellman equation: Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]. When the state space is huge or continuous, the Q-table cannot be stored; DQN parametrizes the Q-function as Q(s,a;θ) to approximate the Q-table, turning the RL problem into a function-fitting problem. This function approximation method enables the algorithm to generalize to unseen states, but also introduces challenges such as training instability.

Experience Replay: Experience Replay is a key technique in DQN for breaking data correlation. In RL, the consecutive experiences (state, action, reward, next state) generated by the agent are strongly correlated; using them directly to train a neural network leads to catastrophic forgetting and training instability. Experience Replay stores experiences in a fixed-size buffer and randomly samples mini-batches from it to update the network, thereby breaking temporal correlation and making the data closer to independent and identically distributed. Additionally, Experience Replay improves data efficiency, as each experience can be sampled and used multiple times.

Target Network: The Target Network is a mechanism in DQN for stabilizing the training target. In standard Q Learning, the target value r + γ max_a' Q(s',a') depends on the current Q-function, which means the target keeps changing as parameters update, creating a "moving target" problem. DQN introduces a target network Q(s,a;θ^-) with the same architecture but fixed parameters to compute the target value, and periodically copies the main network's parameters to the target network. This keeps the target stable for a period, reducing training oscillations and improving convergence.

Code Implementation: Key Snippets and Logic

The note provides code snippets showing the core steps of the DQN training loop. Below is a simplified pseudocode framework that reflects the main logic flow:

# Initialize main network Q and target network Q_target
# Initialize experience replay buffer D
for episode in range(num_episodes):
    state = env.reset()
    while not done:
        # Select action using epsilon-greedy policy
        action = select_action(state, epsilon)
        next_state, reward, done = env.step(action)
        # Store transition in D
        D.push((state, action, reward, next_state, done))
        # Randomly sample a batch from D
        batch = D.sample(batch_size)
        # Compute target Q-values
        targets = compute_targets(batch, Q_target)
        # Compute loss and update main network
        loss = mse_loss(Q(batch.states, batch.actions), targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        state = next_state
    # Periodically update target network
    if episode % update_freq == 0:
        Q_target.load_state_dict(Q.state_dict())

The author specifically points out in the note that how target Q-values are computed depends on whether variants like Double DQN are used. In standard DQN, the target Q-value is typically reward + gamma * max(Q_target(next_state)), where gamma is the discount factor. Although short, the code snippet covers key stages such as action selection, experience storage, sampling, loss computation, and network updates.

Mathematical Foundations: Bellman Equation and Loss Function

DQN's mathematical foundation is the Bellman optimality equation. For a state-action pair (s, a), the optimal Q-value satisfies:

Q*(s, a) = E[r + gamma * max_a' Q*(s', a')]

where the expectation is over random transitions and rewards. DQN uses a neural network Q(s, a; theta) to approximate Q* and learns the parameters theta by minimizing the mean squared Bellman error (MSBE):

L(theta) = E[(r + gamma * max_a' Q(s', a'; theta^-) - Q(s, a; theta))^2]

Here theta^- are the target network parameters, periodically copied from theta. The note explains why a target network is needed: if the same network is used to compute both target and prediction, the target drifts as parameters update, causing training instability. Fixing the target network for a period mitigates this issue. Additionally, the note mentions the specific computation of gradient descent—the gradient of the loss with respect to theta—and emphasizes the need to stop gradient propagation when computing the target.

Mean Squared Bellman Error (MSBE): MSBE is the commonly used loss function in DQN, defined as the expected square of the difference between the target Q-value and the current Q-value. Its expression is L(θ) = E[(r + γ max_a' Q(s',a';θ^-) - Q(s,a;θ))^2]. Minimizing MSBE is equivalent to making the Q-function satisfy the Bellman optimality equation. In practice, the expectation is approximated using mini-batches sampled from the experience replay buffer. During gradient descent, only the current Q-network parameters θ are differentiated; the target network parameters θ^- are treated as constants, i.e., gradient propagation is stopped. Although this semi-gradient method does not theoretically guarantee convergence, it performs well in practice.

Discount Factor (γ): The discount factor γ∈[0,1] balances the importance of immediate and future rewards. When γ is close to 0, the agent focuses more on immediate rewards; when γ is close to 1, the agent values long-term returns more. In DQN, the choice of γ affects the algorithm's convergence and final policy, and typically needs to be tuned for the specific task.

Summary and Further Learning

This one-page note presents DQN's three pillars compactly: the conceptual leap from tables to functions, the code-level Experience Replay and Target Network, and the mathematical Bellman equation and MSBE loss. For beginners, this kind of note that combines theory, implementation, and formulas helps quickly build an overall understanding. The author states in the Reddit post that the note maintains a brief, coherent style and includes code snippets and necessary mathematical explanations. If you want to understand DQN deeply, it's recommended to implement a simple version after reading the note—for example, training in the CartPole environment—to intuitively experience how the algorithm runs.

One-line Summary

A one-page DQN note from Reddit outlines the concepts, code, and mathematical foundations of this classic deep RL algorithm.

Paragraph Summary

Based on a one-page DQN note on Reddit, this article systematically reviews the core content of Deep Q-Network (DQN). It first contrasts traditional Q Learning's tabular method with DQN's neural network function approximation, noting that DQN solves continuous state space problems via neural networks and introduces Experience Replay and Target Network to stabilize training. Then, it presents pseudocode for the DQN training loop, covering action selection, experience storage, sampling, loss computation, and network updates. Finally, it explains the Bellman optimality equation and the Mean Squared Bellman Error (MSBE) loss from a mathematical perspective, and clarifies the role of the target network. The article concludes that the note provides compact introductory material for beginners by integrating concepts, code, and math, and recommends hands-on implementation to deepen understanding.