ClashRoyaleAi: An Open-Source Clash Royale RL Simulator with PPO Agents in Practice

Clash Royale is a real-time, two-player, partially observable card battle game with no official API available for programmatic access — making it an exceptionally challenging environment for reinforcement learning (RL) research. A developer recently open-sourced the ClashRoyaleAi project on Reddit, rebuilding the game's battle engine as a headless deterministic simulator in C++17 and transforming this popular mobile game into a reproducible RL training platform.

Rebuilding a Deterministic Battle Engine from Scratch

The most fundamental contribution of this project is a ground-up reimplementation of Clash Royale's combat logic as a programmable simulation environment. The developer wrote the core engine in C++17 and exposed it to Python via pybind11, ensuring compatibility with mainstream RL ecosystems.

The simulator covers all cards available in-game as of July 2026 — 132 standard cards and 41 Evolutions — and guarantees determinism and seedable randomness. Both properties are critical for RL research: determinism ensures identical inputs always produce identical outputs, enabling reproducibility; seeded randomness makes controlled A/B testing possible.

Performance is equally noteworthy. A complete match runs in approximately 10 milliseconds on a single laptop CPU core, and the snapshot() interface can fork any game state in 7–20 microseconds. This ultra-low-cost state forking capability is the technical foundation for lookahead search and paired A/B testing.

Gymnasium Interface and Observation Space Design

The environment follows the Gymnasium API standard, allowing seamless integration with mainstream RL frameworks. Its observation space is a 13,977-dimensional float vector, with one key design choice being the inclusion of the opponent's card play history — directly addressing the game's partial observability and enabling the agent to make inferences from opponent behavior.

The action space is also carefully structured: the agent selects a card and then chooses a placement from 612 grid positions, with legality masks constraining illegal moves. This structured action definition mirrors the real game's mechanics while reducing the cost of exploring invalid actions.

Gymnasium (formerly OpenAI Gym) is the de facto standard interface for RL, defining a unified contract between environments and agents: reset() initializes the environment and returns the initial observation, while step(action) executes an action and returns a new observation, reward, termination flag, and more. The biggest benefit of conforming to this standard is ecosystem compatibility — frameworks like Stable-Baselines3, RLlib, and CleanRL work out of the box without environment-specific adapters. Legality masks are an important engineering detail in action space design: a boolean vector of the same length as the action space, they set the logits of illegal actions to negative infinity before the policy network's softmax output, ensuring the agent never "tries" to play a card not in hand or place a unit in the river. This significantly improves sample efficiency by not wasting exploration budget on guaranteed-invalid operations.

Agent Architecture: From PPO to Self-Play Leagues

On the agent side, the project uses a recurrent PPO approach with a CNN + LSTM architecture totaling approximately 1.85 million parameters, running on CPU only. Compared to solutions requiring large GPU clusters, this lightweight design is far more accessible for individual researchers to reproduce.

The training strategy builds a multi-layered learning curriculum:

• Curriculum learning: A "teacher" opponent scores candidate card plays via simulation and commands 16 ladder meta decks, progressively guiding the agent's development;
• PFSP self-play league: Prioritized Fictitious Self-Play organizes agent-vs-agent matchups to prevent self-play from degenerating into uninformative games.

PPO (Proximal Policy Optimization) is the most widely used on-policy RL algorithm today. Its core idea is to clip policy update magnitudes to prevent training collapse from overly large update steps, striking a good balance between sample efficiency and implementation simplicity. Recurrent PPO adds an LSTM layer to standard PPO, enabling the agent to maintain hidden state across timesteps — particularly necessary for Clash Royale, where the opponent's hand and intentions are hidden information that must be inferred from historical sequences. PFSP (Prioritized Fictitious Self-Play) originates from DeepMind's StarCraft research. Its key innovation lies in the word "prioritized": rather than randomly selecting historical opponent versions to play against, it preferentially matches the current agent against historical versions with win rates close to 50%, preventing the agent from repeatedly beating already-mastered weaker versions and ensuring continuous effective learning signal.

Experimental Results and the Value of Lookahead Search

The project's preliminary results highlight the dramatic impact of search on policy performance. Adding 1-ply lookahead search on top of the policy network boosted the win rate from 0.625 to 0.944 (160 paired games, 95% CI: +0.24 to +0.40). This is a remarkable improvement, validating the classic "search + learning" combination in game-playing tasks.

The developer then attempted to distill the search gains back into the policy network itself — using value-distribution targets combined with DAgger, ultimately yielding a +0.045 win rate improvement (1,600 paired games, p = 0.007). While modest in magnitude, the result is statistically significant, demonstrating that search capability can be partially internalized into the network.

The developer is candid that the current agent is not yet strong, with a full training run from scratch still in progress.

1-ply lookahead search enumerates all legal actions from the current state, uses the simulator to expand one step forward, evaluates each successor state with a value function, and executes the highest-valued action — essentially upgrading the policy network to a "policy-guided single-step minimax." The dramatic win rate jump (0.625→0.944) indicates that the current policy network makes many errors correctable in a single step, and also suggests the value function's estimate quality is already sufficiently reliable. DAgger (Dataset Aggregation) is an imitation learning algorithm: it has the expert policy (here, the "search-augmented policy") provide labels on the actual state distribution visited by the student policy, rather than training only on expert trajectories, thereby mitigating the distribution shift problem in behavior cloning. Distilling search results back into the network via DAgger essentially teaches the network to "internalize" one-step reasoning, capturing partial benefits without requiring explicit search at inference time.

A Transparent Record of Engineering Decisions

One particularly valuable aspect of this project is its complete documentation of the pitfalls encountered during development. The docs/DECISIONS.md file records every major course correction, including:

• A reward hack — the agent exploited a loophole in the reward function;
• A biased self-play null hypothesis;
• A discount horizon shorter than the actual game length.

For RL practitioners, this kind of "failure log" can be just as valuable as the successful results themselves. It exposes those subtle yet fatal traps that emerge when training agents in real environments.

The project is open-sourced on GitHub (github.com/itzik123/ClashRoyaleAi), with an 87-second demo video at the top of the repository. The developer notes that most of the card logic was built with assistance from Ambash, and that AI coding tools were used as a pair programming assistant throughout — itself a vivid snapshot of how individual developers today leverage AI to complete complex projects.

Summary

ClashRoyaleAi demonstrates how an individual researcher can transform a commercial game with no official API into a serious reinforcement learning research platform. From the deterministic engine, structured observation and action spaces, to the combination of recurrent PPO, lookahead search, and self-play leagues, and through to its transparent engineering decision log, this project offers a highly valuable open-source reference for developers looking to practice RL in complex game environments.