Implementing Reinforcement Learning from Scratch in Python: A Complete Guide from Beginner to Advanced

Introduction: Why Build Reinforcement Learning from Scratch

Recently on Reddit, a developer shared their first project—a reinforcement learning (RL) program implemented entirely from scratch in pure Python—and asked the community for improvement suggestions. This kind of practice is extremely common among AI learners and highly valuable. While mature frameworks like Stable-Baselines3 and Ray RLlib are readily available, writing the core logic of reinforcement learning from scratch often helps learners truly understand the mathematical principles and engineering details behind the algorithms.

The reason reinforcement learning requires ground-up implementation for deep understanding is that it differs fundamentally from supervised learning. Supervised learning relies on pre-labeled datasets, where the model's goal is to fit a mapping from inputs to outputs. Reinforcement learning has no ready-made "correct answers"—the agent must discover optimal behavior through interaction with the environment. This introduces several unique challenges: delayed rewards (the quality of a current action may not become apparent until many steps later), the exploration-exploitation dilemma (whether to try new actions or stick with known good ones), and non-stationarity (changes in the agent's policy alter the distribution of data it collects). These characteristics make it difficult to build genuine intuition by merely calling framework APIs.

Stable-Baselines3 is a high-quality RL algorithm library built on PyTorch, providing standard implementations of mainstream algorithms like PPO, SAC, and TD3 with well-structured code that has been extensively tested and verified. Ray RLlib is an industrial-grade framework designed for distributed scenarios, supporting multi-agent training and large-scale parallel environment sampling. These frameworks dramatically lower the barrier to application, but their high level of abstraction also means users can easily overlook low-level details—such as the timing of gradient clipping, how advantage functions are computed, or the sampling strategy of experience buffers. This is why implementing from scratch remains an irreplaceable learning method.

This article uses this topic as an entry point to systematically review the key considerations when implementing reinforcement learning from scratch, and provides a practical progression roadmap to help beginners upgrade their code from "it runs" to "it runs well and correctly."

Review of Core Reinforcement Learning Components

Before discussing improvements, it's essential to verify that your implementation covers the core components of RL. A complete reinforcement learning system typically contains the following parts.

The Agent-Environment Interaction Loop

The essence of reinforcement learning is an agent learning optimal policies through trial and error in an environment. At its core is a loop: the agent observes a state, selects an action, and the environment returns a reward and a new state. If your first program is built from scratch, start by checking whether this interaction loop is clear and reusable.

A good practice is to follow the OpenAI Gym (now Gymnasium) interface conventions: even for custom environments, provide reset() and step(action) methods that return (observation, reward, done, info). The benefit is that you can seamlessly integrate with standard algorithm libraries and evaluation tools in the future.

OpenAI Gym was released in 2016 and quickly became the de facto standard environment interface in reinforcement learning. In 2022, since OpenAI was no longer actively maintaining it, the community forked the Gymnasium project (maintained by the Farama Foundation) as its successor. The core design philosophy of this interface is minimalist uniformity: any environment only needs to implement the reset() and step() methods to interface with any algorithm library. This standardization has greatly promoted reproducibility in RL research and cross-environment comparison of algorithms. Currently, thousands of third-party environments follow this interface, covering robotics control, games, financial trading, and more.

Representing Value Functions and Policies

Beginners most commonly start with tabular methods like Q-Learning or SARSA, using a two-dimensional array or dictionary to store state-action values Q(s, a). This is an excellent starting point for understanding Temporal Difference (TD) Learning. The core update formula is worth studying repeatedly:

Q(s, a) ← Q(s, a) + α [r + γ·max Q(s', a') − Q(s, a)]

Here α is the learning rate and γ is the discount factor. If your program has correctly implemented this update logic, congratulations—you've mastered the most fundamental and important component of reinforcement learning.

Temporal Difference (TD) learning is one of the most central ideas in reinforcement learning, formally introduced by Richard Sutton in 1988. It combines the advantages of Monte Carlo methods (learning from complete episodes) and dynamic programming (using successor state estimates for bootstrapping). Q-Learning is an off-policy TD control method that directly learns the optimal action-value function without depending on the current policy; SARSA is an on-policy method whose update uses the next action actually chosen by the agent. The core difference between the two lies in whether the update target takes the max (Q-Learning) or the value of the actually executed action (SARSA), leading to different convergence properties and safety characteristics—SARSA learns more conservative but safer policies in environments like cliff walking.

Six Improvement Suggestions for RL Beginners

Regarding "how to improve your first RL program," the community typically offers advice in the following directions.

1. Introduce an Exploration-Exploitation Balance Mechanism

Many beginner implementations cause the agent to converge prematurely to suboptimal policies, rooted in insufficient exploration. Consider introducing an ε-greedy policy with epsilon decay over training: a high exploration rate early on encourages experimentation, while a low rate later focuses on exploitation. This small change often significantly improves learning performance.

Beyond ε-greedy, several other exploration strategies are worth knowing. Boltzmann exploration (softmax action selection) assigns selection probabilities based on the relative magnitudes of Q-values, offering more nuance than ε-greedy. UCB (Upper Confidence Bound) methods borrow from multi-armed bandit theory, giving exploration bonuses to infrequently tried actions. More advanced methods like Curiosity-driven Exploration use prediction error as intrinsic reward, guiding agents to actively explore novel states, performing well in sparse-reward environments.

2. Standardize Hyperparameter Management

Centralize management of hyperparameters like learning rate, discount factor, exploration rate, and number of training episodes, rather than hardcoding them throughout the code. Use a configuration dictionary or argparse command-line arguments. This facilitates hyperparameter experiments—RL is extremely sensitive to hyperparameters, and systematic tuning ability is key to advancing.

The hyperparameter sensitivity problem is particularly pronounced in reinforcement learning. Research shows that the same algorithm under different hyperparameters can exhibit performance differences of orders of magnitude. A learning rate that's too large causes Q-values to diverge, while one too small leads to extremely slow convergence; the discount factor γ determines how much the agent values long-term rewards and needs careful adjustment for different task horizons. At advanced stages, tools like Weights & Biases or Optuna can be introduced for systematic hyperparameter search and experiment tracking.

3. Add Training Process Visualization

Printing numbers alone makes it difficult to judge whether the algorithm is truly learning. Use Matplotlib to plot episode reward curves over time. A curve that steadily rises and gradually plateaus is a visual sign of successful training; dramatic oscillation or stagnation suggests parameter adjustment or logic checks are needed.

Beyond reward curves, several important metrics deserve monitoring: episode length reflects changes in the agent's survival ability; the mean and variance of Q-values can diagnose overestimation problems; the exploration rate decay curve helps confirm whether the exploration strategy is working as expected. Using moving averages (such as the average of the last 100 episodes) smooths out noise from randomness, making trends clearer. TensorBoard is also a powerful visualization tool that supports real-time monitoring of multiple metrics.

4. Transition from Tabular Methods to Function Approximation

When the state space grows large (e.g., continuous states), tabular methods break down. The next step is to try approximating Q-values with simple linear functions or neural networks—stepping toward Deep Q-Network (DQN). This step introduces important techniques like Experience Replay and Target Networks.

Experience Replay is one of DQN's key innovations, first introduced by DeepMind in their 2013 Atari paper and refined in the 2015 Nature paper. The core idea is to store the agent's interaction experiences (s, a, r, s') in a fixed-size buffer, then randomly sample mini-batches for gradient updates during training. This breaks the temporal correlation between consecutive experiences, making neural network training more stable. The Target Network is another stability trick: maintaining a parameter-lagged copy of the network to compute TD targets, synchronized every fixed number of steps or through soft updates (Polyak averaging), preventing training divergence from "chasing itself." Subsequent improvements include Double DQN (addressing overestimation), Dueling DQN (separating state value and advantage functions), and Prioritized Experience Replay (preferentially sampling experiences with large TD errors).

5. Ensure Experiment Reproducibility

Fix random seeds (random.seed, numpy.random.seed) and record the configuration and results of each experiment. RL experiments have high randomness, and reproducibility is a basic requirement for serious research and engineering.

Sources of randomness in reinforcement learning are diverse: environment state transitions may be stochastic, exploration strategies introduce random action selection, neural network parameter initialization varies, and even minor differences in floating-point arithmetic across hardware can cause results to diverge. Research by Henderson et al. in 2018 showed that performance improvements in many published RL papers could not be reproduced with different random seeds. Therefore, good practice is to run multiple seeds (typically 5-10) and report mean and standard deviation, while using version control to record code state and pinning dependency library versions.

6. Write Clean Code Structure and Tests

Separate the environment, agent, and training loop into independent modules, and write unit tests for key functions (e.g., verifying that Q-value updates match expectations). Good engineering habits make future extensions much easier.

A recommended project structure typically includes: an envs/ directory for environment definitions, agents/ for algorithm implementations, utils/ for logging and visualization tools, configs/ for hyperparameter configurations, and tests/ for unit tests. Key test cases include: verifying Q-value convergence to theoretical optima in deterministic environments, checking experience replay buffer read/write logic, and verifying that ε decay proceeds as expected. This modular design also facilitates swapping different algorithms or environments for comparative experiments.

Progression Roadmap: From Toy Project to True Mastery

Completing your first RL program is just the starting point. Here's a recommended growth path:

• Phase 1: Thoroughly understand Q-Learning / SARSA, validating on classic environments like FrozenLake and CartPole.
• Phase 2: Learn policy gradient methods (REINFORCE, Actor-Critic), understanding the difference between value-based and policy-based approaches.
• Phase 3: Implement or reproduce modern algorithms like DQN and PPO, comparing your implementation's performance against Stable-Baselines3.
• Phase 4: Read Sutton & Barto's classic textbook Reinforcement Learning: An Introduction to solidify your theoretical foundation.

Policy gradient methods directly parameterize the policy function π(a|s;θ) and maximize expected cumulative reward through gradient ascent. REINFORCE is the most basic policy gradient method, using complete episode returns as weights for gradient estimation, but with extremely high variance. The Actor-Critic architecture reduces variance by introducing a value function (Critic): the Actor outputs the policy, the Critic evaluates the current state's value, and both update cooperatively. PPO (Proximal Policy Optimization) is currently the most widely used policy gradient algorithm, proposed by John Schulman et al. at OpenAI in 2017. It limits policy update magnitude by clipping the objective function, achieving an excellent balance between implementation simplicity and training stability. PPO is widely applied in game AI, robotics control, and the RLHF (Reinforcement Learning from Human Feedback) training of ChatGPT, making it one of the most commonly used RL algorithms in industry today.

Sutton & Barto's Reinforcement Learning: An Introduction (second edition, 2018) is considered the "bible" of the reinforcement learning field, with its electronic version freely available on the authors' website. The book starts from multi-armed bandits and systematically covers dynamic programming, Monte Carlo methods, temporal difference learning, function approximation, and policy gradients. Combined with David Silver's UCL reinforcement learning course videos and Spinning Up in Deep RL (OpenAI's educational project) for code practice, these form the most recommended self-study resource combination for RL currently available.

Conclusion

Implementing reinforcement learning from scratch in pure Python is something worth celebrating. It forces you to confront every detail of the algorithm rather than treating it as a black box. For the Reddit developer and all learners at the same stage, perhaps the best advice is: first make it correct, then make it clear, and finally make it powerful. Continuous iteration, systematic experimentation, and solid theory—combining all three, you'll quickly grow from "wrote an RL program" to "truly understanding reinforcement learning."

Key Takeaways