reinfors Adds CarRacing Environment: Rust Backend Delivers 20x Reinforcement Learning Speedup

How a Rust Backend Accelerates Reinforcement Learning

A long-standing bottleneck in reinforcement learning (RL) research often isn't the model itself, but the environment sampling speed. The training paradigm of RL is fundamentally different from supervised learning: supervised learning datasets are prepared in advance, while RL requires agents to interact with the environment in real-time to generate training data. Each interaction step involves environment state updates (physics simulation, collision detection, reward calculation, etc.), and these computations typically run on the CPU. When environment stepping speed is too slow, GPU training resources are forced to wait for data collection on the CPU side, wasting compute power—in complex environments, GPU utilization can drop as low as 10%-30%.

The open-source project reinfors addresses exactly this pain point by rewriting environment simulation and sampling logic with a Rust backend, while leaving neural network components entirely under the control of the caller's Python code. Rust is an ideal choice for replacing Python in environment simulation due to its unique language features: zero-cost abstractions mean high-level code compiles to performance approaching hand-written C; the ownership system eliminates data races at compile time, enabling safe multi-threaded parallelism without locks; and the absence of a garbage collector means no unpredictable pauses. In contrast, Python's Global Interpreter Lock (GIL) makes true multi-threaded parallelism nearly impossible, and the overhead from its dynamic typing and interpreted execution is amplified many times over in high-frequency loops.

In the latest v0.3.0 release, reinfors adds the classic CarRacing environment. CarRacing is a classic continuous control task from OpenAI Gym where an agent must drive a car on a randomly generated track, taking 96×96 RGB pixel images as input and outputting continuous actions for steering, throttle, and braking. This environment is important because it simultaneously involves visual perception (requiring convolutional neural networks to process pixel input) and continuous control (requiring policy networks to output continuous values), making it a classic testbed for validating end-to-end RL algorithm capabilities. This port not only fills a highly requested community need (previously requested by user u/blimpyway), but more importantly introduces a modular rendering layer—its rendering layer involves 2D physics simulation, track generation, vehicle dynamics, and pixel-level rendering, meaning adding new rendering-based game environments in the future will be quite straightforward.

Performance: ~20x Single-Core Stepping Speed Improvement

Based on the benchmark data published by the author, reinfors shows quite impressive improvements in environment stepping speed:

Single-Threaded Stepping Comparison

• Apple M1 Max: ~20x faster than Gymnasium (3,850 vs 195 steps/sec)
• AMD EPYC EC2 instance: ~14x faster (2,069 vs 148 steps/sec)

Interestingly, the author maintains a relatively rigorous approach to methodology: benchmarks take the median of three 30-second trials, execution order is alternated, warm-up phases are discarded, both sides use single-threaded bare stepping loops, and machine and software environment information is automatically printed and recorded by script. This transparent testing approach avoids the common "cherry-picking good results" style of performance marketing.

Multi-Threaded Parallel Stepping

With 10 worker threads on M1 Max, the reinfors engine achieves 8,000+ steps/sec. However, the author also honestly notes that this figure is not an "apples-to-apples" comparison with Gymnasium's AsyncVectorEnv, and therefore should not be simply equated to a speedup ratio in parallel scenarios. It's worth noting that Gymnasium's AsyncVectorEnv implements parallelism via Python's multiprocessing module, with each sub-environment running in a separate process where inter-process communication (IPC) serialization/deserialization overhead is non-trivial; meanwhile, reinfors' multi-threaded approach benefits from Rust's lack of a GIL, allowing threads to share memory with zero-copy and extremely low communication overhead.

collect_stream: Overlapping Training and Sampling Execution

The true value of reinfors may go beyond raw stepping speed. Its core design, collect_stream, allows native data collection to overlap with GPU training on the Python side—this is its default operating mode. This Actor-Learner overlap architecture breaks the serial dependency of traditional RL training, similar to pipelining in computer architecture: while the Learner processes batch N on the GPU, the Actor is already preparing batch N+1 on the CPU.

In other words, while your PyTorch or other framework performs backpropagation on the GPU, Rust worker threads continue collecting new experience data in the background, and the trainer never has to idle waiting for the collector. In large-scale distributed RL systems (such as DeepMind's IMPALA and OpenAI's Rapid), this architecture is standard, but typically requires complex engineering to handle data synchronization, policy lag (off-policy correction), and other issues. reinfors encapsulates this complexity in the Rust backend, allowing users to achieve pipelined training with just a simple Python API.

The author points out that the aforementioned speedup multiples actually underestimate the impact on overall training, as they only measure raw stepping speed without accounting for time savings from training overlap. While similar actor-learner overlap can be achieved with native Gym, it typically requires writing complex custom logic or introducing third-party frameworks, which is why it wasn't included in this benchmark comparison.

Practical Usage: PPO Training Loop Code Example

The reinfors API design embodies the philosophy of "Rust handles sampling, Python handles networks." Here's a typical PPO training configuration:

import numpy as np
import reinfors as rf

engine = rf.Engine(
    game=rf.games.CarRacing(),   # pixel observations, shape (3, 96, 96)
    reward=rf.Reward(tile=1000.0, step=-0.1, off_playfield=-100.0),
    policy=rf.policies.Ppo(),
    learner=rf.learners.Ppo(gamma=0.99, lam=0.95),
    n_games=64,    # parallel episode slots
    n_threads=8,   # full parallel configuration
)

def infer(obs: np.ndarray):
    # Your network, any framework: batch observations in, (logits, values) out
    # e.g., a torch CNN running on GPU
    ...

with engine.collect_stream(collect_size=4096, infer=infer) as stream:
    for update in range(200):
        batch = next(stream)  # Rust workers keep collecting, you keep training
        # Standard clipped PPO update

PPO (Proximal Policy Optimization) is a policy gradient algorithm proposed by OpenAI in 2017, widely adopted as one of the most popular RL algorithms due to its simplicity and stable performance. Its core idea is to limit the magnitude of each policy update by clipping the objective function, preventing policy collapse. In the code, gamma=0.99 is the discount factor controlling how much the agent values future rewards; lam=0.95 is the λ parameter for GAE (Generalized Advantage Estimation), used to trade off bias and variance in advantage function estimation. collect_size=4096 means a policy update is performed after collecting 4096 steps of experience, and this batch size directly affects gradient estimation stability.

The network inference function infer is entirely user-provided, with no framework constraints—whether PyTorch, JAX, or any other framework can be plugged in. Batches contain standard PPO fields including obs, actions, advantages, returns, behavior_log_probs, and the specific update logic is entirely in the user's hands. The behavior_log_probs record the log-probabilities of actions output by the policy during data collection, which is key to PPO's computation of importance sampling ratios, used to constrain the difference between old and new policies from becoming too large.

Details to Note Before Using reinfors

Before adopting reinfors, there are several technical details worth considering:

Implementation Differences: reinfors' CarRacing and Gymnasium's CarRacing-v3 are two independent implementations of the same game. There are subtle differences between them, so trajectories and floating-point-level physics cannot be directly transferred between the two. These differences mainly stem from numerical precision in the physics engine, random number generator implementations, and subtle differences in collision detection algorithms—in chaotic systems (such as vehicle dynamics), small initial differences accumulate and amplify over time steps. The good news is that pixel-trained agents can run in the Gym environment by transposing observations (HWC → CHW). reinfors directly outputs observations in CHW format (Channel-Height-Width), which is the input format expected by PyTorch convolutional layers; standard Gymnasium outputs HWC format (Height-Width-Channel), the default format for NumPy and OpenCV. When deploying a trained agent to a standard Gymnasium environment, a simple np.transpose(obs, (2, 0, 1)) completes the format conversion.

Performance Data Boundaries: The author is quite clear about the boundary conditions of performance numbers—20x is the M1 Max result, ~14x on EC2; the 8,000 steps/sec parallel figure is not a strict apples-to-apples comparison. This restrained presentation is uncommon in open-source project promotion and enhances the project's credibility. The performance gap between M1 Max and EC2 instances may stem from Apple Silicon's unified memory architecture providing cache-friendliness advantages—environment simulation involves frequent access to many small data structures and is extremely sensitive to memory latency.

Summary: A New RL Engineering Approach with Rust+Python

reinforss represents a noteworthy approach to reinforcement learning engineering: delegating compute-intensive, highly optimizable environment simulation to Rust, while leaving the flexible and ever-changing neural network components to the Python ecosystem. This hybrid architecture philosophy already has successful precedents in ML engineering—NumPy's underlying layer is C/Fortran, PyTorch's operator core is C++/CUDA, while user-level flexibility is maintained by Python. reinfors extends this philosophy to RL's unique environment simulation domain, achieving efficient interoperability between the two languages through PyO3 (Rust's Python binding library).

For scenarios where environment stepping speed is a research bottleneck, this architecture can significantly shorten experiment iteration cycles. The project author has also expressed willingness to help the community port other environments limited by stepping speed. Potential beneficiary scenarios include multi-agent environments (requiring simulation of dozens or even hundreds of agent interactions per step), high-fidelity physics simulation environments, and research requiring massive parallel experiments for hyperparameter search.

It's currently installable via pip install reinfors, with source code hosted on GitHub (github.com/jeepjeepjeep/reinfors). For developers engaged in reinforcement learning research—especially those struggling with sampling efficiency—this is an open-source tool worth trying.