Training AI in the Browser: A Deep Dive into Clash Royale Reinforcement Learning

A developer has distilled their Clash Royale AI agent into a miniature version that can be trained directly in the browser — letting curious onlookers watch a neural network learn defensive strategy from scratch in real time. This project is more than a fun demo; it's a rare exercise in making reinforcement learning (RL) transparent, with every gradient written out in plain sight.

A Mini Agent That Learns in the Browser

After the developer published their full Clash Royale AI, the comments quickly filled with questions about how it actually learned. Rather than explaining PPO in replies, they built an interactive lab page: on screen, an attacking unit charges toward your tower, and you have exactly one card to stop it. Where you place it and when both matter enormously.

Visitors can try defending five times themselves, then watch a small neural network learn the same task from scratch. Crucially, every attempt runs through the team's real simulator — an engine compiled from C++ to WebAssembly, the same one used to train the full agent. This means the browser mini-experiment and the large-scale training share the same physics core, not a simplified approximation.

Under the Hood: Every Gear of Reinforcement Learning

The developer is admirably explicit about the RL design, which is where the project's educational value really shines:

• State: The position where the attacking unit is deployed
• Action: A valid grid cell plus a delay between 0 and 5 seconds
• Reward: The fraction of tower damage prevented, compared to doing nothing
• Algorithm: REINFORCE with a per-spawn baseline, totaling 5,629 weights

What's most impressive is the engineering restraint: the entire learner is roughly 350 lines of pure JavaScript, with every gradient hand-coded and zero dependency on any machine learning library. This "build from bare metal" approach strips away the black box of frameworks, making the learning process genuinely readable and inspectable.

The chart also includes a "theoretical optimum" reference line, computed by brute-forcing every grid cell and delay combination. This line gives you an immediate, intuitive sense of how far the current policy is from optimal — something most RL demos conspicuously lack.

REINFORCE is one of the earliest policy gradient algorithms, introduced by Williams in 1992. Its core idea is simple: if an action was followed by a high reward, increase its probability; otherwise, decrease it. Unlike value-function methods such as Q-learning, REINFORCE directly differentiates through the policy itself without explicitly estimating state values. The "per-spawn baseline" is a variance-reduction technique — by subtracting a baseline (typically a running average of past rewards), it reduces the noise in gradient estimates and stabilizes learning. Without it, pure Monte Carlo sampling produces variance so high that convergence becomes nearly impossible. PPO (Proximal Policy Optimization) is an advanced evolution of REINFORCE that clips update magnitudes to prevent drastic policy shifts — it's the go-to choice for production-grade RL agents, and indeed what the full Clash Royale AI uses with a recurrent network. The fundamental difference comes down to stability and sample efficiency; the mini version deliberately uses the simpler REINFORCE so that every computational step can be fully exposed.

Giant vs. Cannon: A Classic RL Trap

The developer recommends starting with the Giant versus Cannon matchup. Intuitively, placing the Cannon directly in the attacker's path seems like the obvious move — but this approach only earns about 75% of the optimal score. The true optimal solution is to place the Cannon in the center lane, drawing the Giant into the attack range of both towers — a counterintuitive but far smarter positioning.

This exposes a classic RL challenge: exploration. The developer found that with a constant entropy bonus, the network got stuck in the suboptimal "place on the path" solution in 5 out of 6 runs (tested across 3 random seeds and 2 batch sizes). When the entropy coefficient was annealed from 0.1 down to 0.005 over the first 10,000 attempts, the failure rate dropped to just 1 in 6. This is the configuration currently live on the page. The detail vividly illustrates how exploration scheduling can be decisive in escaping local optima.

An entropy bonus is a common technique in RL for encouraging exploration. A policy's entropy measures how "random" it is — higher entropy means the agent is more likely to try different actions rather than committing to a single choice. Maintaining high entropy early in training helps the agent break out of suboptimal habits formed at the start; gradually annealing the entropy coefficient as training progresses then guides the policy toward convergence and determinism. Annealing is essentially a scheduling strategy borrowed from simulated annealing: broad exploration at high "temperature," fine-grained exploitation at low "temperature." This connects directly to the local optimum problem: if exploration is reduced too early, the policy "locks in" to the first solution that looks good and never discovers globally better action combinations. The developer's data — annealing cutting the stuck rate from 5/6 to 1/6 — is a clear, quantified validation of this theory in a real system.

A Deliberately Unsolved Challenge

One matchup in the project is intentionally left unsolved: Battle Ram versus Valkyrie. Its optimal solution requires precise placement in specific cells at precise timings, and every configuration the developer tried ultimately settled for a "safe corner" approach, achieving only about 55% of the optimal score.

The developer openly throws this unsolved problem to the community: "If you find a learner that cracks it, I genuinely want to see it." This openness transforms the project from a one-way demo into a collaborative research invitation, while honestly capturing the inherent difficulty RL faces when precise, sparse action sequences are required.

The Distance Between Mini and Full

The developer repeatedly stresses this is just a "scaled-down model." The full agent plays entire matches, manages a hand of 4 cards plus elixir resources, uses recurrent PPO, and trains for days on CPU.

But the core loop is identical: act → let the engine score it → update. This simple closed loop is the shared skeleton connecting the mini experiment to the full-blown agent. For anyone who wants to understand how reinforcement learning actually learns, watching this curve climb in a browser is far more intuitive than reading an algorithm paper.

Both the lab and the full project's code are open source, hosted in the GitHub repository, and are accessible to any developer who wants to dig into RL's internals firsthand.

WebAssembly (WASM) is a binary instruction format that runs in browsers at near-native speed, officially standardized by the W3C in 2019. Developers can compile code written in C, C++, Rust, and other languages into WASM and execute it in the browser's sandboxed environment — with performance far exceeding traditional JavaScript. For this project, compiling the C++ game simulator to WebAssembly serves a dual purpose: it ensures that the browser-side and server-side physics rules are completely identical, avoiding the misleading simplifications of a "demo version"; and it makes high-frequency simulation sampling (RL requires massive amounts of interaction to generate sufficient training signal) feasible in the browser without performance bottlenecks that would slow or distort the experiment.