Training a PPO Agent to Farm D2R's The Pit in a Rust-Written Simulator

A Reddit user recently shared his hardcore RL project: training a PPO (Proximal Policy Optimization) agent to automatically farm "The Pit" in Diablo II: Resurrected (D2R). Training isn't done directly in the game—it runs in a headless simulator the author wrote by hand in Rust. Training takes about 20 minutes on a single CPU, and the agent has roughly 475,000 parameters. This project brings reinforcement learning down to a specific scenario in a classic ARPG, and both the approach and the engineering trade-offs are quite interesting.

Why Choose "The Pit" as the Training Ground

The Pit is a classic level-85 area in D2R, with dense monsters and a good drop pool, making it the top choice for many players farming gear. For RL, the appeal of this scene lies in: a clear objective (clear monsters, grab drops, don't die), a moderate episode length, and a state and action space much smaller than the entire map. The author also mentioned that he initially added an extra reward for "clearing the level" but later removed it—this shows he prefers the agent to align directly with the ultimate goal of "farming efficiency" rather than being led astray by intermediate goals.

From a task design perspective, this scenario is naturally suited to using "farming rate" as the reward: the more kills and drops per unit time, and the fewer deaths, the better the policy. This reward alignment approach is more direct than artificially designed hierarchical rewards and is also easier to transfer to other farming scenarios.

PPO (Proximal Policy Optimization) is one of the most commonly used policy gradient algorithms in reinforcement learning today, proposed by OpenAI in 2017. Its core idea is to limit the deviation of the new policy from the old policy at each update (by clipping the importance sampling ratio), thereby balancing sample efficiency and training stability. Compared to value-function methods like DQN, PPO directly optimizes the policy and is better suited to continuous or high-dimensional action spaces; compared to TRPO, it is simpler to implement and computationally lighter. This is precisely why PPO has become the default baseline algorithm in scenarios such as game AI and robot control, and it is also the key prerequisite for the small 475,000-parameter network in this example to converge in 20 minutes on a single CPU.

In the context of reinforcement learning, a "headless simulator" refers to a lightweight runtime environment that does not render a graphical interface and only retains game logic and state transitions. By eliminating the overhead of graphics rendering and user interaction, a headless simulator can generate training samples rapidly and in batches, and supports parallelization and time acceleration. For closed-source commercial games like D2R, building your own headless simulator also avoids the compliance risks of directly injecting into the game process.

Reward Design: Driving Policy with an Economic Calculus

The reward function the author provided is straightforward: -1 point per second, points for kills and items according to value, and -1000 points for death. In other words, the only thing the agent needs to learn is "to farm as much value as possible as quickly as possible while staying alive."

The cleverness of this design lies in making the time cost explicit. Many game AIs use "level completion rewards" or "kill counts" as the main signal, but the core of farming gameplay is actually efficiency. The -1 per second penalty naturally eliminates dawdling, detouring, and ineffective movement policies; the -1000 death penalty provides a sufficiently strong risk signal to prevent the agent from dying in pursuit of drops. The author mentioned that there was an early level-clear reward but it was later removed, which also indirectly shows that intermediate rewards tend to make the policy "farm points" rather than "farm maps."

Engineering Implementation: Rust Headless Simulator + Warm Start

The training environment is not the game itself, but a headless simulator written by the author in Rust. This step is crucial: D2R is a closed-source commercial game, and running RL directly in the game is both slow and involves compliance issues, whereas in a self-built simulator you can accelerate and sample in batches arbitrarily. The performance advantages brought by Rust make training in 20 minutes on a single CPU possible.

Another detail is "warm start": the author did not train from scratch, but continued training from a network built for an older version. This is very common in RL engineering—reusing an existing policy can significantly shorten convergence time, and it also shows that the author had already iterated in this direction before. The network scale of about 475,000 parameters is lightweight for PPO, consistent with the positioning of "trainable on a single CPU."

The author explicitly stated that he has no plans to use the solution online and just finds RL fun. This sense of boundaries is commendable: offline simulator training does not violate the game's terms of service and does not disrupt other players' experience.

"Warm start" in reinforcement learning engineering usually refers to initializing the training of a new task with the weights of an existing policy network, rather than random initialization. The opposite is training from scratch (cold start). The value of warm start lies in: the existing network has often already learned some reusable feature representations or behavior patterns (such as movement and attack timing), and the new task only needs fine-tuning on this basis, thereby greatly reducing the required samples and training time. In this example, the author continued training from a network built for an older version, precisely leveraging the similarity of scenes and mechanics between D2R versions. It should be noted that warm start can also bring negative transfer—if the old and new tasks differ too much, the old policy may become a starting point for local optima, so in practice it is often combined with learning rate adjustments or partial layer freezing.

What This Project Demonstrates

From a single point of view, this is a fun project where a player uses RL to reproduce "farming" behavior; from a methodological point of view, it demonstrates several reusable lessons: replacing the real environment with a self-built simulator, aligning the ultimate goal with efficiency-based rewards, reducing training costs with warm start, and matching compute constraints with a lightweight network. These lessons are valuable references for developers working on game AI, automated testing, and even robotic process optimization.

Of course, there is inevitably a gap between the simulator and the real game (monster AI, drop randomness, network latency, etc.), and the author did not elaborate on simulation accuracy, which is also a common limitation of such projects. But as a practice of "RL is fun," it walks through the complete chain from environment setup to reward design, and it is informative enough.

FAQ

Q: Can this agent be used on online Battle.net? A: The author explicitly stated that he has no plans for online use; training is entirely conducted in an offline simulator.

Q: How much compute was used for training? A: About 20 minutes on a single CPU, warm-started from an older-version network, with about 475,000 parameters.

Q: Why remove the level-clear reward? A: The author initially added a level-clear reward but later removed it, so that the reward more directly reflects farming efficiency.

One-line Summary

A Reddit user trained a PPO agent to automatically farm D2R's The Pit using a Rust headless simulator, in about 20 minutes on a single CPU.

Paragraph Summary

A Reddit user shared his reinforcement learning project: using the PPO algorithm to train an agent to automatically farm "The Pit" in Diablo II: Resurrected. Training is not done in the game itself, but in a headless simulator the author wrote by hand in Rust, taking about 20 minutes on a single CPU with a network of only about 475,000 parameters. The reward function is designed around farming efficiency: -1 point per second, points for kills and items according to value, and -1000 points for death, with the early level-clear reward removed to align with the ultimate goal. The author also used warm start, continuing training from an older-version network to reduce convergence costs. The project is for offline simulation only, and the author explicitly has no plans for online use, demonstrating reusable lessons in game AI such as self-built simulators, efficiency-based rewards, and warm start.