Reinforcement Learning Takes On Getting Over It: Distributed Training Strategies Behind 63% Progress

A developer on Reddit is using deep reinforcement learning (DRL) to tackle Getting Over It with Bennett Foddy (commonly known as "Getting Over It"), a game notorious for its difficulty. Currently, his agent has climbed from the bottom to 63% progress, but is stuck at the anvil section. The project is open-sourced on GitHub and Hugging Face, and he is publicly seeking community help. This is not just a fun attempt at game AI—it also reflects the typical challenges and engineering solutions of DRL in sparse-reward, long-horizon planning tasks.

Project Status: The 63% Milestone and the Anvil Bottleneck

According to developer miwate's Reddit post, the current agent reaches 63% height from the bottom of the game without checkpoints. The game is segmented into multiple parts, with a "specialist model" trained for each part—a divide-and-conquer strategy aimed at reducing the difficulty of training on long-horizon tasks. However, the anvil section has become a new obstacle—this area requires precise physical interaction and continuous decision-making, placing higher demands on exploration efficiency and reward design.

The developer admits that training is currently done entirely on CPU, because in his tests, GPU did not bring any speedup. This may be related to the computational characteristics of the game environment itself: physics simulation and state updates may become the bottleneck, rather than the neural network's forward/backward propagation.

Specialist Model in reinforcement learning refers to a model trained separately for a specific subtask or local region of the environment, as opposed to a "generalist model" that attempts to solve all tasks with a single model. Divide-and-conquer training simplifies the learning problem by reducing the state space and action space each specialist needs to cover, and is often used for long-horizon, sparse-reward tasks. However, combining specialist models faces transition problems: policies from different specialists may be discontinuous at boundaries, requiring switching mechanisms or joint fine-tuning. In addition, no checkpoints means the agent must execute continuously from start to finish—any mid-way failure nullifies all previous progress, placing higher demands on the robustness and generalization of specialist models.

Technical Breakdown: Divide-and-Conquer Training and the CPU Choice

Segmenting the game into multiple parts and training specialist models separately is a common approach to handling sparse rewards. Each specialist only needs to focus on a local objective, reducing learning difficulty. But this also brings new problems: how to ensure continuity between specialist models? The developer did not disclose details, but from the "no checkpoints" statement, the agent needs to pass through multiple regions continuously in one run, which is a huge test of model robustness.

Regarding CPU training, the developer mentioned that "using GPU did not make training faster." This may be because environment stepping consumes a large amount of time, and the neural network is small in scale, so the parallel advantages of GPU cannot be fully utilized. For individual researchers, this is actually good news—DRL experiments can be conducted without expensive hardware.

Sparse Reward is a classic challenge in reinforcement learning: the agent receives zero or minimal reward at the vast majority of time steps, with feedback only when a specific goal is achieved (such as reaching a new height). This makes credit assignment difficult—the agent struggles to determine which action sequences contribute to the final reward, resulting in extremely low exploration efficiency. Getting Over It is a typical sparse-reward environment, because only height changes provide reward signals, and the intermediate process lacks guidance. Common mitigation methods include reward shaping, curiosity-driven exploration, hindsight experience replay (HER), etc., but all require careful task-specific design.

Environment Stepping refers to the process in the reinforcement learning training loop of executing actions, advancing physics simulation, updating states, and computing rewards. In physics-based game environments, stepping often involves complex collision detection and rigid body dynamics calculations, and a single step may take far longer than the neural network's forward/backward propagation. When environment stepping becomes the bottleneck, the parallel computing advantages of GPU are difficult to leverage, because training speed is limited by the CPU-side simulation speed. This is one of the fundamental reasons why the developer observed no GPU speedup—especially when the neural network is small, the computational graph overhead on GPU may actually be worse than CPU.

Community Collaboration: Open Source and PR Invitations

The project code has been made public on GitHub (miwate/getting-over-it-rl), with models hosted on Hugging Face (miiwater/goi-rl). The developer is writing CONTRIBUTING.md and explicitly welcomes "beginner code and vibe coded PRs," promising to take time to review them. This open attitude lowers the barrier to contribution, and also suggests the project is still in its early stages, needing more hands to solve computational resources, fine-tuning, and code optimization problems.

In the Reddit post, the developer directly calls for "help with online compute, fine-tuning, and PRs." For RL enthusiasts interested, this is an opportunity to participate in a real project—whether providing compute, improving reward functions, or optimizing the training pipeline, all could help push the agent past the anvil.

Why This Project Is Worth Following

Getting Over It is known for its tricky physics engine and frustration—difficult for human players, and even more challenging for AI. It requires the agent to make precise controls in a continuous action space, and rewards are extremely sparse (feedback only when reaching new heights). This project thus becomes a touchstone for testing the robustness of DRL algorithms. If successful, its divide-and-conquer training and specialist model combination ideas may transfer to other long-horizon decision-making tasks.

Currently, 63% progress has already proven the feasibility of the approach, but the stagnation at the anvil section also exposes its limitations. Can the community help break through? We'll see.

(Note: This article is based on the Reddit post and public repository information, and does not include game screenshots.)