Training a PPO Agent to Play Temple Run on a Phone: Real-Device Record Hits 16,782 Meters

A developer shared on Reddit his complete project of using reinforcement learning to let an AI beat Temple Run. What makes this project special is that the AI doesn't learn visually from game screen pixels — instead, it directly reads game memory states and directly calls the game's own methods to control the character. Behind it is a high-speed environment implemented in C++ that can complete 10 million training steps in just 6 minutes in a simulator.

Why Choose Temple Run for Getting Started with Reinforcement Learning?

The author mentioned that he undertook this project to truly master reinforcement learning (RL). Compared to benchmark environments like CartPole or Atari that have been thoroughly explored, using a game from his childhood as a practice target gave him more motivation and made it easier to persist through the entire pipeline.

The bigger challenge lies in platform differences. Mobile games have more complex runtime environments, input methods, and state reading than traditional desktop RL environments, which turns the entire project from "getting the algorithm to work" into a complete system integration exercise: it requires solving a series of engineering problems including memory reading, action injection, simulator replication, and real-device calibration.

The author chose PPO (Proximal Policy Optimization) as the training algorithm. PPO is one of the most widely used policy gradient methods in reinforcement learning in recent years, known for stable training and relatively friendly hyperparameter tuning, making it suitable for projects that need to build a training pipeline from scratch.

PPO (Proximal Policy Optimization) was proposed by OpenAI in 2017 and belongs to the policy gradient family under the Actor-Critic architecture. Its core idea is to limit the difference between old and new policies at each update, using a clipped surrogate objective function to prevent policy updates from being too large and causing training collapse. Compared to TRPO, PPO is simpler to implement and has lower computational overhead; compared to vanilla policy gradients, it has higher sample reuse and more stable training. These characteristics have made PPO the default choice in large projects such as OpenAI Five and the RLHF stage of ChatGPT, and it is also one of the most common algorithms for getting started with reinforcement learning today. Understanding PPO's clipping mechanism and advantage function estimation is a necessary prerequisite for reading the training pipelines of such projects.

CartPole is the most classic introductory reinforcement learning environment in OpenAI Gym: an inverted pendulum is balanced by applying left or right force, with only a 4-dimensional state and 2 actions, typically trainable to a stable policy with a few hundred lines of code in tens of seconds. The Atari benchmark includes dozens of pixel-level games where states are high-dimensional images requiring convolutional networks, and is the standard test suite for evaluating the generalization ability of deep RL algorithms. These two types of environments are widely used because they have unified interfaces, high reproducibility, and comparable results. Temple Run in this project, however, has neither a standard interface nor runs on a mobile platform, making it a typical "non-standard environment" with engineering difficulty far exceeding the above benchmarks.

Direct Memory Reading, Direct Method Calls: An Agent Design That Bypasses Pixels

The most noteworthy technical decision in this project is the Agent's interface design.

State input doesn't rely on pixels. The agent reads game states directly from game memory through a Frida gadget. This means it gets structured internal data rather than raw frames that require convolutional networks to interpret. The advantage is high sample efficiency and fast training convergence; the disadvantage is that the Agent is strongly coupled to this specific game's memory layout.

Action output doesn't rely on simulated touchscreen. The agent executes actions through the game's own methods rather than simulating screen taps or swipe gestures. This eliminates noise from input latency and touch coordinate errors, making the training signal cleaner.

Frida plays a key role here. Frida is a dynamic instrumentation tool commonly used for reverse engineering and runtime analysis. The author used it as a bridge for memory reading and action injection, effectively turning the game into a "quasi-API environment" that can be programmatically queried and controlled.

This "white-box" approach is not uncommon in game RL projects, but fully implementing it on mobile and achieving results is no small engineering feat.

Frida is an open-source Dynamic Binary Instrumentation (DBI) framework, originally designed specifically for mobile reverse engineering, supporting Android, iOS, Windows, macOS, and other platforms. It injects a JavaScript runtime into the target process, allowing developers to use JS scripts to hook arbitrary functions, read/write memory, and call object methods at runtime without recompiling or modifying the target program. In game reverse engineering and assistant development, Frida is often used to locate key functions, export memory structures, and bypass anti-debugging measures. In this project, the author uses the Frida gadget (an embedded library that can be loaded into the target process) as a bridge to expose the game process's internal state and operations as a programmable interface, transforming an ordinary mobile game into an environment that RL agents can directly interact with.

Headless Simulator and C++ Acceleration: 10 Million Steps in 6 Minutes

Training efficiency is another highlight of this project.

With Claude's assistance, the author built a headless simulator based on the game's internal mechanics and calibrated it using parameters measured on a real phone. This is crucial: if the simulator deviates too much from real-device behavior, the policy will fail when transferred to the phone.

The simulator has a C++ version that can complete 10 million training steps in about 6 minutes. This throughput means experiments can be iterated quickly without waiting for lengthy real-device trial and error. For individual developers, the ability to run experiments quickly often directly determines whether a project can keep moving forward.

From an engineering perspective, the project structure can be broken down into four layers: the game memory interface layer, the calibrated simulator, the PPO training pipeline, and real-device deployment validation. Each layer has its own independent failure modes, and any layer going wrong would zero out the final result.

"Headless" refers to a program running without a graphical interface, retaining only core logic computation and skipping all rendering, sound, UI drawing, and other overhead unrelated to decision-making. For reinforcement learning training, graphical rendering is often one of the biggest performance bottlenecks — rendering a single frame can take orders of magnitude longer than physics simulation and state updates. A headless simulator strips away all these components, keeping only game mechanics relevant to policy learning such as physics, spawning, collision, and scoring, thereby compressing per-step time to the microsecond level. This also explains why this project can complete 10 million steps in 6 minutes: that works out to about 28,000 steps per second, a throughput nearly impossible to achieve in a real game environment with graphics. The Sim-to-Real field commonly adopts this approach: train at high speed in a headless environment first, then return to the real environment for validation.

Real-Device Testing: Median 4,736 Meters, Best 16,782 Meters

The training results were ultimately validated on a real phone. The author provided data from 14 runs:

• Median score: 4,736 meters
• Best score: 16,782 meters

The gap between the median and the best record is quite significant, indicating that the policy's performance varies considerably across different runs. This is common in game RL — randomly generated terrain, item distribution, and speed increase curves all affect individual run performance, and the median reflects policy stability better than the best score.

The author set 16,782 meters as the public challenge target and open-sourced the GitHub repository (matthieucs/TempleRun-RL), welcoming others to try to beat this score.

What Reference Value Does This Project Have for People Working on Game AI

First, interface design affects final results more than algorithm choice. This project uses standard PPO with no special algorithmic innovations, but eliminates a large amount of environmental noise through memory reading and native method calls. For projects with limited resources, optimizing the interface often yields higher returns than switching algorithms.

Second, simulator calibration is a key step in Sim-to-Real. The author explicitly mentions that the simulator was calibrated with real-device measurement data, and this is not a step that can be skipped. Many personal projects perform well in simulators but collapse on real devices, and this is usually the reason.

Third, a high-speed simulator makes the experimental pace feasible. 10 million steps in 6 minutes means dozens of experiment rounds can be run in a single day, and this iteration speed is the prerequisite for tuning a good policy.

Fourth, using a "game you actually enjoy playing" as a learning project is an effective way to self-motivate. This project revolves around the author's own interests from motivation to execution, ultimately producing a work with clear quantitative results that others can reproduce and challenge.

If you also have a mobile game you've been playing for years and want to find a concrete goal to practice reinforcement learning, this project's approach provides a fairly complete reference template: white-box interface + calibrated simulator + fast training + real-device validation.

Sim-to-Real is a core challenge shared by robotics and game AI: policies trained in simulators often suffer sharp performance degradation when transferred to real environments, because simulators cannot fully reproduce real-world noise, latency, parameter errors, and dynamic variations. Common mitigation methods include Domain Randomization (randomly varying simulator parameters during training to enhance policy robustness) and System Identification (using real data to inversely fit simulator parameters). This project chose the latter — the author explicitly calibrated the simulator with real-device measurement data to make the simulator's state transitions and speed curves as consistent as possible with the real device. Although the method is not novel, for personal projects, the rigor of this step often directly determines whether the policy can truly "land."

One-line Summary

A developer used PPO + a Frida white-box interface + a calibrated simulator to make an AI run as far as 16,782 meters in Temple Run.

Paragraph Summary

A developer open-sourced a complete project using reinforcement learning to beat Temple Run. Its core feature is that it does not use pixel input or touch simulation, but instead uses Frida dynamic instrumentation to directly read game memory states and call native game methods to execute actions, thereby eliminating visual noise and input latency. The author used Claude to help build a headless simulator based on the game's internal mechanics and calibrated parameters with real-device data; the C++ version completes about 10 million training steps in roughly 6 minutes. The algorithm uses standard PPO with no algorithmic innovation, with results coming mainly from interface design optimization and Sim-to-Real alignment. In 14 real-device tests, the median was 4,736 meters and the best was 16,782 meters; the clear gap between median and best reflects that policy stability still fluctuates. The project has been open-sourced on GitHub with the challenge target made public.