CUDA-Accelerated Rocket League RL Simulator: Training 20B Steps on a Single RTX 4090

In reinforcement learning training, the most time-consuming part is often not the model computation, but the environment simulation. One developer spent nearly two years porting the physics simulation of Rocket League entirely to the GPU, using CUDA to build a massively parallel environment and achieving order-of-magnitude improvements in training efficiency.

What Is CARL? Bringing Rocket League Physics into CUDA

This project is called CARL (CUDA-Accelerated Rocket League). The core idea is to keep all simulation state in GPU memory, completely eliminating the overhead of host-device data transfer during training. The developer was inspired by PureJaxRL—which had previously implemented a similar massively parallel environment approach in JAX—and decided to build a dedicated GPU-parallel environment for Rocket League, a task he was personally interested in.

Previously, the community had RocketSim, a high-performance CPU simulator already integrated into RLGym for reinforcement learning training. CARL is not meant to replace RocketSim, but to explore another path: using CUDA to run physics logic in parallel on the GPU, allowing thousands of games to advance simultaneously. According to the developer, his bot trained on CARL can seamlessly transfer to the real game, with physics behavior essentially identical.

CUDA (Compute Unified Device Architecture) is NVIDIA's general-purpose parallel computing platform and programming model, allowing developers to directly call thousands of GPU cores to execute non-graphics computing tasks. In reinforcement learning, environment simulation is typically executed serially on the CPU, whereas CUDA lets each thread independently handle one environment instance, thereby advancing thousands of games simultaneously. Host-device transfer refers to data copying between the CPU (host) and GPU (device) over the PCIe bus, whose bandwidth is far lower than GPU memory. As the number of parallel environments increases, each simulation step needs to synchronize observations, actions, and rewards, and the transfer overhead quickly becomes a bottleneck. CARL keeps all state in GPU memory precisely to eliminate this overhead entirely.

Performance Data: 21 Hours to Run the Equivalent of 21 Years of Game Time

The developer released a set of training data that is enough to demonstrate the efficiency of this approach. On a single RTX 4090, using 8192 parallel environments and frameskip=4 for training, about 20B decision steps (corresponding to 80B physics ticks, or roughly 21 years of game time) took just over 21 hours of wall-clock time.

From the training curve, the bot shows clear improvement in aerial control. The developer also included a capability demo video showing the bot's aerial maneuvering after training. However, he also joked that if he were controlling it himself, a slight maneuver might save it—implying that although the bot performs well, it is still some distance from top human players.

The key to this efficiency gain lies in eliminating the host-device transfer bottleneck. Traditional CPU simulators need to synchronize data between CPU and GPU at every step, and as the number of parallel environments grows, this overhead becomes the main bottleneck. CARL keeps all state on the GPU, allowing thousands of environments to truly run in parallel.

frameskip is a common parameter in reinforcement learning environments, indicating how many underlying physics engine simulation steps advance for each decision action the agent performs. For example, with frameskip=4, for every action the bot takes, the physics world runs 4 consecutive ticks before returning the next observation frame. This both lowers the decision frequency and shortens the training sequence length, while allowing the physics simulation to maintain relatively high precision. Therefore, the 20B decision steps corresponding to 80B physics ticks in the article is a direct result of frameskip=4. Choosing an appropriate frameskip is a key trade-off between training efficiency and policy quality: too large and the bot loses fine-grained control ability; too small and training costs rise sharply.

Project Status and Future Plans

The developer says CARL is currently "feature-complete" and decided to open source it so the community can give feedback. He also admits that he learned a lot about GPU programming and optimization, but still has much more to learn, and welcomes suggestions from the community.

The upcoming TODO list includes:

• Built-in training package API for targeted training
• Speed and physics consistency benchmarks
• Additional tests and other engineering best practices
• Exploring JAX and TensorFlow integration

One detail: the evaluation of physics parity is still relatively preliminary. The developer says the bot can seamlessly transfer to the real game, which is a strong signal, but more systematic comparison work has not yet been completed.

JAX is a high-performance numerical computing library developed by Google, supporting automatic differentiation, vectorization, and JIT compilation, and can run efficiently on GPU/TPU. PureJaxRL is a reinforcement learning library built on JAX, whose core feature is implementing end-to-end training in a purely functional way, so that the entire training loop can be JIT-compiled and parallelized at scale without frequent CPU-GPU interaction. CARL was inspired by it, but chose the route of hand-writing physics kernels in CUDA, because Rocket League's physics logic is relatively complex and expressing it in JAX may have performance or flexibility limitations. Integrating JAX/TensorFlow means that in the future users can directly call the CARL environment within these frameworks, thereby reusing the existing training algorithm ecosystem.

Community Response and Significance

After being posted on Reddit, this project attracted considerable attention. Massively parallel environments are an important direction in current reinforcement learning research, with work such as PureJaxRL and Isaac Gym driving this trend. CARL's value lies in bringing this idea to a concrete and challenging game task, and verifying seamless transfer from GPU simulation to the real game.

For researchers and engineers doing reinforcement learning training, CARL offers an idea: when environment simulation becomes the bottleneck, moving physics logic onto the GPU for parallel execution may bring greater benefits than optimizing the model itself. Of course, this approach also has its scope of applicability—it requires the physics logic to be simple enough to parallelize efficiently, and the task itself to be suitable for large-scale environment sampling.

The developer plans to release more demos later. As the features are improved, CARL may become a powerful tool for Rocket League RL training.

One-line Summary

A developer used CUDA to move Rocket League physics simulation onto the GPU, completing training equivalent to 21 years of game time in 21 hours.

Paragraph Summary

CARL is an open-source project that fully ports Rocket League physics simulation to the GPU. Its core idea is to keep all simulation state in GPU memory, eliminating host-device transfer overhead and supporting thousands of parallel environments. On a single RTX 4090, using 8192 parallel environments and frameskip=4, about 20B decision steps (80B physics ticks, roughly 21 years of game time) were completed in just 21 hours of training. The trained bot can seamlessly transfer to the real game with essentially identical physics behavior, but systematic physics parity evaluation is still incomplete. Inspired by PureJaxRL, the project is now feature-complete and open source, with future plans to add a training package API, benchmarks, and JAX/TensorFlow integration, offering a practical approach to reinforcement learning training by moving physics logic onto the GPU when environment simulation becomes the bottleneck.