Turning Rain World into a Reinforcement Learning Environment: A Gymnasium + SB3 Implementation

From Classic Teaching Environments to Real Games

For developers just getting into Reinforcement Learning (RL), standard environments like CartPole and FrozenLake are almost a rite of passage. They're simple in structure, easy to understand, and well-suited for verifying algorithm logic. But these environments lack real-world complexity and rarely inspire sustained exploration.

To be specific, CartPole's state space has only 4 continuous variables (cart position, velocity, pole angle, and angular velocity), with an action space of just 2 discrete actions (push left/push right). FrozenLake is a simple grid world where the number of states equals the number of tiles. These environments share a common trait: they converge within hundreds to thousands of training steps, making it difficult to distinguish subtle algorithmic differences. Real-world control problems—whether robot locomotion, autonomous driving, or game AI—typically involve high-dimensional observations (such as pixel inputs), continuous action spaces, delayed rewards, and environmental non-stationarity. This enormous gap between teaching environments and real-world scenarios often leaves learners feeling lost after completing introductory tutorials.

So one developer did something quite creative: they transformed their favorite game, Rain World, into a fully-featured reinforcement learning environment. This served as both their first deep dive into RL and a contribution of an entirely new, highly challenging RL testbed for the community.

The project is open-source, with the repository available at: https://github.com/XRTG074/reinforced-rain-world

Why Choose Rain World as a Reinforcement Learning Environment

Rain World is an indie game renowned for its ecosystem complexity, featuring one of the most intricate and diverse ecosystems in video games. The game's creatures each possess independent behavioral logic, predator-prey relationships, and habitats. The player controls a small creature that must survive in this harsh yet elegant world.

From a technical perspective, Rain World was independently developed by Joar Jakobsson over approximately 7 years. Its ecosystem contains dozens of creatures with independent AI behavior trees. Each creature has its own perception range, threat assessment logic, food chain position, and social behavior. The game's physics engine is based on procedural animation, where characters' bodies consist of multiple physically-constrained nodes, and movement is the result of real-time physics simulation rather than pre-set animations. This means every movement by the player (or AI agent) is a direct interaction with the physics engine—jump height depends on velocity and angle at takeoff, and climbing requires precise timing coordination. This physics-simulation-based movement system is structurally closer to real robot control problems than most platform games.

This level of environmental complexity is precisely what RL research craves. Compared to classic environments with limited state spaces, Rain World offers rich physical interactions, varied terrain, and mechanics requiring fine-grained motor control. Previously, almost no one had built an RL environment based on Rain World, and this gap is exactly where this project's value lies.

The Challenge of Movement Control

In the demo videos, the developer showcases the agent learning to "walk" and "climb vertical poles." The agent's climbing motion looks rather "bizarre" because it's attempting to master an advanced movement technique called "pole-boosting."

This detail is quite revealing. In reinforcement learning, agents don't solve problems the way humans do—instead, guided by the reward function, they explore all sorts of unexpected strategies. As long as a strategy yields higher returns, it gets continuously reinforced. This "emergent behavior" is one of RL's most fascinating properties.

Emergent behavior has many classic cases in RL research history: in OpenAI's hide-and-seek experiments, agents learned to exploit physics engine boundary conditions to pass through obstacles; DeepMind's football AI discovered that sliding tackles from the side were more efficient than frontal defense. The appearance of these behaviors is closely related to reward function design (reward shaping). Reward shaping is one of the most skillful aspects of RL engineering—overly sparse rewards lead to exploration difficulties, while overly dense rewards may guide agents to find shortcuts rather than learn genuine skills. In complex environments like Rain World, designing reward functions that guide learning direction without overly constraining the policy space is an open problem with significant research value.

Technical Implementation: Gymnasium Wrapper and Stable-Baselines3 Integration

From an engineering perspective, the most practical aspect of this project is that the developer added a Gymnasium wrapper to the environment.

Gymnasium is the community-maintained successor to OpenAI Gym and has become the de facto standard interface for RL environments. Through unified APIs like reset(), step(), observation_space, and action_space, any Gymnasium-compliant environment can seamlessly integrate with mainstream RL algorithm libraries.

More specifically, Gymnasium defines a standard interaction protocol for RL environments: the reset() method returns an initial observation and info dictionary; step(action) receives an action and returns a five-tuple (observation, reward, terminated, truncated, info); observation_space and action_space declare the data types and value ranges for observations and actions respectively (supporting space types like Box, Discrete, MultiDiscrete, MultiBinary, etc.). The value of this protocol lies in decoupling environments from algorithms: environment developers only need to implement the specified interface, algorithm developers only need to program against the interface, and neither party needs to understand the other's internal implementation. The Gymnasium ecosystem currently covers hundreds of environments including Atari games, MuJoCo physics simulations, and robot control, forming the largest interoperable environment collection in the RL field.

Leveraging this, the developer made the environment directly compatible with algorithms from Stable-Baselines3. Stable-Baselines3 (SB3) is one of the most popular RL algorithm implementation libraries, built on PyTorch with a modular design that separates policy networks, value functions, replay buffers, and training loops into independent components. Its built-in algorithms cover major paradigms: PPO (Proximal Policy Optimization) works for both continuous and discrete action spaces and is known for training stability; SAC (Soft Actor-Critic) is the current benchmark for continuous control tasks; DQN and its variants (Double DQN, Dueling DQN) specialize in discrete action scenarios. SB3 also provides standardized callback mechanisms for training monitoring, model saving, and early stopping, as well as integration with TensorBoard/Weights&Biases for experiment tracking.

This means any developer who clones this repository can plug mature algorithms into the Rain World environment at virtually zero cost and start training and experimenting immediately. For environments like Rain World that involve fine-grained motor control, PPO and SAC are typically the go-to algorithms.

Engineering Challenges: From Game to RL Environment

It's worth noting that converting a commercial game into an RL environment involves multiple engineering challenges. First is game loop modification—most games run rendering loops at fixed frame rates, while RL training needs to advance simulation steps as quickly as possible, typically requiring a headless mode without rendering for tens-of-times speedup. Second is state extraction—obtaining structured information like character position, velocity, and surrounding environment from within the game, which may require reverse engineering or modding the game code. Third is action mapping—translating numerical actions output by RL algorithms into input commands the game recognizes. Finally, there's determinism and reproducibility—RL training requires environments to support random seed setting to reproduce experimental results. Rain World is built on the Unity engine and has an active mod community, which provides a relatively friendly foundation for environment modification.

Standard Tech Stack Lowers the Barrier to Entry

For learners looking to get hands-on practice, the combination of "standard interface + mature algorithm library" dramatically lowers the barrier to entry. You don't need to implement PPO from scratch, nor do you need to build environment interaction logic yourself—just focus on reward design, hyperparameter tuning, and result analysis. And the complex physics and movement challenges that Rain World provides make experimental results far more interesting than CartPole.

Insights for Reinforcement Learning Learners

Although this project is modest in scale, it embodies several ideas worth learning from:

Interest-driven learning is more sustainable. Transforming a game you love into an experimental environment naturally provides motivation for continued investment, and makes it easier to understand the complexity behind the environment.

Complex environments reveal an algorithm's true capabilities. Classic environments are great for debugging, but real challenges come from high-dimensional states, continuous actions, and subtle physical interactions. Rain World's movement control problems are an excellent testbed for this.

Following community standard interfaces is crucial. Through the Gymnasium wrapper, the developer gave their personal project reusable, collaborative value, and contributed a new open resource to the community.

Conclusion

From CartPole to Rain World, this leap represents not just an increase in difficulty, but a spirit of exploration: placing abstract algorithms into real, complex environments to test them. The developer has expressed a warm welcome to community feedback, which is precisely the most admirable attitude in open-source projects.

For developers looking to go deeper in reinforcement learning, an RL environment based on an indie game like this might be the ideal starting point for transitioning from educational toys to real-world challenges.