Open-Source Clash Royale RL Simulator: Lessons from a C++ High-Speed Engine and PPO Agent

A Clash Royale Simulator Built for Reinforcement Learning

In game AI research, the lack of efficient and controllable training environments is often the biggest bottleneck. A recent open-source project aims to tackle this problem head-on: a Clash Royale simulator built from scratch in C++, designed specifically for reinforcement learning (RL) training scenarios.

As shared on the Reddit community, the simulator's most impressive performance metric is approximately 10 milliseconds per game. This means that within a training loop, an agent can complete an enormous number of matches in an extremely short time, dramatically reducing sample collection overhead. For RL algorithms that depend on large-scale interaction data, this kind of speed advantage is decisive.

Forkable State: Opening the Door to Search and Planning

Another core feature of the project is forkable state. This design allows the complete game state to be copied at any moment, after which different action sequences can be explored from that branch.

This capability is highly valuable for modern RL and planning algorithms. Whether it's Monte Carlo Tree Search (MCTS) — which requires repeatedly simulating future branches — or training approaches that involve counterfactual reasoning, both demand that the environment efficiently save and restore state. Compared to replaying a match from scratch each time, forkable state lets researchers explore multiple paths from the same decision point, enabling more precise evaluation of different strategies.

The low-level C++ implementation is equally critical here — state copying only avoids becoming a performance burden if it's lightweight enough. This explains why the author chose to build the engine core in C++ rather than the more approachable Python.

Monte Carlo Tree Search (MCTS) is a tree-search algorithm based on random sampling, best known for its successful application in Go-playing programs (it was a core component of AlphaGo). The basic idea is to repeatedly execute four steps starting from the current state — selection, expansion, simulation, and backpropagation: select a promising node in the tree, expand new actions from that node, simulate the game to a terminal state randomly or heuristically, then backpropagate the result to update value estimates for each node. This process requires a large number of independent simulations from the same node, making it highly dependent on the environment's ability to save and restore state. If every simulation had to replay from the start of the match, the computational cost would grow exponentially. Forkable state allows each simulation to begin from a snapshot of the target node, dramatically reducing per-simulation overhead and making MCTS practically feasible in high-branching-factor environments like real-time strategy games.

Recurrent PPO: Handling the Partially Observable Game Environment

On the algorithmic side, the project uses recurrent PPO (Proximal Policy Optimization). PPO itself is one of the most widely applied policy gradient algorithms today, known for training stability and relatively straightforward implementation. Adding a recurrent network architecture (such as LSTM/GRU) gives the agent memory of historical information.

This choice directly addresses the fundamental nature of Clash Royale. As a real-time card battle game, players cannot fully observe the opponent's hand, elixir accumulation, or card intentions — making it a classic partially observable environment (POMDP). Pure feedforward networks struggle to make coherent decisions in such environments, while recurrent architectures can encode past observations into a hidden state, helping the agent infer the opponent's hidden information and formulate long-term strategies.

A Partially Observable Markov Decision Process (POMDP) is a more accurate model of real-world decision scenarios: the agent cannot directly access the environment's complete state and must rely on noisy or incomplete observations to infer the current situation. In Clash Royale, this manifests as the opponent's hand composition, current elixir count, and next card being invisible — the agent must reverse-engineer the opponent's strategic intent from their past action sequence. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) are the mainstream recurrent network architectures for handling such problems. They maintain a hidden state vector updated at each timestep, compressing historical observations into a fixed-dimensional memory. Compared to single-step feedforward policy networks, recurrent architectures give the agent the ability to integrate information across timesteps, enabling it to recognize the opponent's card-playing patterns and adjust its own strategy accordingly, rather than merely reacting to the current frame.

The Agent as the Best Tester: Engine Bugs Exposed

One intriguing detail from the project's sharing is that the agent exposed bugs in the simulator itself during training. This is actually a classic phenomenon that recurs throughout RL engineering.

When a sufficiently powerful optimizer is placed inside an environment, it has no understanding of the designer's intent — it simply and tirelessly seeks paths that maximize reward. If the simulator's physics rules, damage calculations, or state transitions contain any exploits, the agent will often find them precisely and exploit them relentlessly. This is known as reward hacking or specification gaming.

From a positive perspective, an RL agent therefore becomes an extremely rigorous automated testing tool. Edge-case action combinations that human testers might never think to try will be systematically covered by the agent across millions of matches. The flaws it exposes in turn push the simulator to more closely approximate the real game's behavior. This cycle of "train → discover bug → fix → retrain" is the unavoidable path to building a reliable simulation environment.

Reward hacking and specification gaming are two closely related core challenges in reinforcement learning. Reward hacking refers to an agent finding a way to obtain high rewards without genuinely completing the task the designer intended — for example, spinning in place in a racing game to continuously accumulate turning bonuses rather than actually completing the track. Specification gaming more broadly refers to an agent satisfying the literal definition of the reward function while violating its design intent. The root cause of these phenomena is that any reward signal expressed as a mathematical function struggles to fully capture all of a human's expectations for a "good outcome." In game simulator scenarios, this problem is further amplified: if the simulator itself deviates from the real game's rules, agents will treat those deviations as exploitable loopholes and amplify them. This is why "adversarial testing" is increasingly valued in simulation environment development — and a trained RL agent naturally fills this role.

What This Means for Game AI Researchers

Taking all these features together, this project provides a relatively complete experimental infrastructure: the high-speed C++ engine solves the throughput problem, forkable state supports search-based algorithms, and recurrent PPO offers a viable baseline for partially observable environments.

For developers hoping to do RL research in complex real-time strategy games, the open-source nature means they can build directly on top of this work rather than constructing an environment from scratch. Domain-specific high-performance simulators like this are often the critical link that enables algorithmic research to move from theory to practice.

It should be noted that the information currently available comes primarily from community sharing. The level the agent ultimately achieves, specific network architecture parameters, and the degree of consistency between the simulator and the real game still await further disclosure in the project documentation. Interested researchers are encouraged to consult the open-source repository directly for first-hand information.