Building a Reinforcement Learning Environment for a Pokémon Roguelike: An Open-Source RL Agent Challenge

A Data Scientist's Open-Source Experiment

Reinforcement Learning (RL) has always been one of the most fascinating and challenging branches of AI. As one of the three major paradigms of machine learning (alongside supervised learning and unsupervised learning), RL's core idea originates from the trial-and-error learning theory in behavioral psychology: an Agent learns optimal behavioral strategies through continuous interaction with an Environment — at each step, it observes the current state, selects an action, and the environment returns a new state along with a numerical reward signal. The agent's goal is to maximize long-term cumulative reward. This framework is formally described by Markov Decision Processes (MDPs), and everything from AlphaGo to RLHF training in ChatGPT is built upon this theoretical foundation.

Recently, a data scientist shared his open-source project on Reddit: a complete reinforcement learning environment built for the browser game Pokelike (a Pokémon-style Roguelike game). He published the GitHub repository and invited developers worldwide to take on the challenge.

Roguelike is a classic game genre named after the 1980 game Rogue, characterized by procedurally generated levels, permadeath mechanics (you start over when your character dies), and turn-based gameplay. This genre, with its high randomness and irreversible decisions, is a natural ideal testing ground for reinforcement learning — agents can't simply memorize fixed routes but must learn generalized strategies to handle each unique random situation.

The core idea behind this project is crystal clear: run the game locally and expose its state and actions directly to the AI agent. Unlike many game AIs that rely on screen image recognition, this environment involves no image processing at all — the agent directly receives structured game state data (such as current Pokémon attributes, HP, inventory items, and other internal variables) and then decides what to do next. In game AI research, this approach stands in stark contrast to pixel-level visual input: the latter requires computationally intensive methods like convolutional neural networks to extract useful information from images (as in DeepMind's famous Atari experiments), while the former bypasses the entire visual perception pipeline, allowing researchers to focus purely on decision-making strategies, significantly reducing computational costs and training difficulty.

What Decisions Does the Agent Need to Make?

In this Pokémon Roguelike, the decisions facing the AI agent are far more complex than you might expect. It needs to make choices during battles, including: where to move on the map, which Pokémon to catch, which items to pick up, when to swap out the active Pokémon, and which moves to learn.

The Long-Term Impact of Early Decisions

The author specifically highlighted an interesting design feature: the map forces players to make decisions very early on. Once you choose a particular node, all other nodes on the same level disappear. This means seemingly insignificant early choices can have far-reaching consequences later in the game. This "path dependency" is precisely what makes RL training tricky — the agent must learn to plan for long-term gains rather than simply chasing immediate rewards. In reinforcement learning theory, this involves the "Credit Assignment Problem": when the final outcome results from a chain of decisions, how do you determine which early decisions played a critical role? This has been one of the core challenges the RL field has been tackling for decades.

Three Challenging Properties of the Environment

From a reinforcement learning perspective, this environment has several properties that make it "more interesting than expected":

• Sparse Reward: Reward signals don't appear frequently, making it difficult for the agent to learn quickly. This is one of the thorniest challenges in reinforcement learning. In dense reward environments, the agent receives feedback at every step (like speed scores at each frame in a racing game), providing ample learning signals. But in sparse reward environments, rewards only arrive upon achieving key objectives (such as defeating a gym leader or earning a badge), with zero reward across the many intermediate steps. This makes it nearly impossible for the agent to distinguish good from bad behavior during early exploration, often requiring techniques like Curiosity-Driven Exploration, Experience Replay, or Hierarchical RL to mitigate the issue.
• State-dependent action space: The available actions change depending on the current state (for example, the moves available in battle depend on the currently active Pokémon). This adds complexity to policy learning because the agent cannot assume a fixed set of action options at every state.
• Delayed rewards: Some decisions only reveal their value several steps later. For instance, catching a Pokémon that seems weak now but has complementary type coverage might only demonstrate its strategic value three battles later.

These properties are precisely a microcosm of many real-world reinforcement learning problems — from robot control to drug discovery, portfolio management to autonomous driving, all face the challenges of sparse rewards, dynamic action spaces, and delayed returns. This also makes this seemingly simple Pokémon game an excellent algorithm testing ground.

Implemented RL Agents and Benchmark Results

The author has already implemented several baseline reinforcement learning agents in the repository and provided current benchmark results. Performance is measured by the "number of badges earned":

Agent	Average Badges
Random Policy	~0.56
Dyna-Q	~0.62
Linear SARSA (81 features)	~1.30
Linear SARSA (100 features)	~1.36

The data shows that the random policy baseline only earns about 0.56 badges, while the classic Dyna-Q algorithm offers only marginal improvement at 0.62. Dyna-Q was proposed by RL pioneer Richard Sutton in 1991 and is a classic method that combines model-based and model-free learning. While learning from real interactions, it also builds an internal environment model (learning state transitions and reward functions), then uses this model for "simulated experiences" to accelerate learning — in plain terms, the agent learns not only from actual gameplay but also practices through "mental simulations." However, in this project, Dyna-Q only slightly outperforms the random policy, likely because the game environment's complexity and vast state space make it difficult to build an accurate internal model.

The real performance gap comes from the two Linear SARSA agents, which achieved 1.30 and 1.36 respectively. SARSA (State-Action-Reward-State-Action) is an on-policy temporal difference learning algorithm that updates value estimates based on the action sequence the agent actually takes, rather than assuming the agent will always choose the optimal future action as Q-learning does. When the state space is too large for tabular storage (which is clearly the case for this game's complex states), function approximation is typically used, with linear function approximation being the simplest form: the state is encoded as a feature vector, and a linear weighting estimates the value of each action. The quality of feature design directly determines the expressiveness of the value function.

The Critical Role of State Representation in RL Performance

The most thought-provoking aspect of these results is that the main difference between the two SARSA agents lies solely in their state representation. The better-performing one uses 100 hand-crafted features, while the other uses only 81. This difference in feature count and design alone led to a noticeable performance improvement.

The author openly acknowledged that this is currently his most interesting direction: "Finding a good state representation seems to make quite a big difference." This confirms a classic insight in the reinforcement learning field — feature engineering and state representation often matter more than the algorithm itself in determining final performance, especially in complex environments where information is abundant but the right representation is unclear.

Before the widespread adoption of deep learning, feature engineering was the make-or-break factor in machine learning projects. In reinforcement learning, the quality of state representation determines whether the agent can effectively distinguish different situations and make appropriate decisions. A good state representation should contain all key information relevant to decision-making while eliminating redundant noise. For example, in Pokémon battles, "the current type matchup advantages," "remaining HP ratio," and "type coverage of available moves" are all potentially valuable features. Even in the era of deep reinforcement learning, how to design input representations (or inductive biases in network architecture) remains one of the core factors affecting final performance. This also hints at an enticing direction: could deep networks automatically learn state representations that surpass the ceiling of hand-crafted features?

A Fully Reproducible Fair Arena

To enable fair comparisons between different agents, the author made an important engineering decision: the environment is fully reproducible. Using the same seed and the same action sequence produces an identical game progression.

This design is hugely significant for RL research. Reinforcement learning research has long faced a reproducibility crisis: a widely cited 2018 paper by Henderson et al. pointed out that experimental results in many RL papers are highly sensitive to random seeds, hyperparameters, and even minor differences in code implementation, meaning that performance improvements reported in papers may not be reproducible by other researchers. This problem continues to plague the community to this day.

The current leaderboard uses 50 fixed seeds, ensuring all agents are tested on the exact same game scenarios. This design eliminates interference from randomness, making performance comparisons more scientifically rigorous and convincing. In open-source RL projects, this is a commendable engineering practice.

Additionally, the entire environment can run completely offline. The game and necessary resources are downloaded during initial setup, after which all computation runs locally with no need for a continuous internet connection. This is crucial for RL training that may require tens or even hundreds of thousands of game simulations — network latency or service instability could severely slow down training progress.

Open Interface Design: Not Limited to Any Specific RL Algorithm

The most appealing aspect of this project is its openness. The interface is intentionally designed to be very simple — you just need to implement a "bot" that receives the current state and returns an action. It is not limited to any specific reinforcement learning algorithm.

Whether you want to try Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), Monte Carlo Tree Search (MCTS), or even hand-code a set of heuristic rules, you can experiment within this framework.

Each of these three methods has its strengths: DQN was proposed by DeepMind in 2013, combining deep neural networks with Q-learning to achieve human-level performance on Atari games for the first time — a milestone in deep reinforcement learning. Its core innovations include the experience replay buffer and target network. PPO was proposed by OpenAI in 2017 as a policy gradient method that ensures training stability by limiting the magnitude of policy changes at each update. Thanks to its simple implementation and robust performance, it has become one of the most popular RL algorithms today — it is also the core algorithm used in the RLHF stage of ChatGPT. MCTS is a core component of AlphaGo, evaluating the value of different actions by running extensive random simulations on a decision tree, making it particularly well-suited for games with clear turn-based and branching structures.

The author provides clear instructions in the repository's GUIDE.md. Developers just need to follow the guide to set up the environment, and if they're satisfied with their results after experimenting, they can submit a Pull Request to the bots/ folder to enter their agent into the leaderboard competition.

An Invitation to the Developer Community

The author originally built this as a "fun little RL project," but now he's more interested in seeing how far the community can push it. He particularly hopes someone can achieve significantly better results through better state representations or different reward functions.

Reward function design (also known as "Reward Shaping") is another core topic in reinforcement learning. The current project uses badge count as the primary reward metric, but developers are free to design more fine-grained intermediate rewards — for example, providing additional positive signals for successfully catching Pokémon, discovering favorable type matchups, or reaching new map areas. Good reward shaping can dramatically accelerate learning, but poor design can lead agents to "game the system" — pursuing reward signals rather than the actual game objective. This phenomenon is known as "Reward Hacking" in the RL literature.

For developers and students looking to get hands-on practice with reinforcement learning, this is a low-barrier, highly engaging opportunity: zero cost to set up the environment, fully reproducible results, and complete freedom in algorithm choice. It provides an intuitive understanding of core RL challenges like sparse rewards, delayed returns, and state representation, while preserving the fun of the game itself.

If you're interested in reinforcement learning, head over to the repository and see if your agent can earn more Pokémon badges.