CUDA-Accelerated Rocket League RL: Device-Resident Simulation Makes Training 100x Faster

In reinforcement learning (RL), the speed of environment simulation often sets the upper bound on training efficiency. While most people are still running game simulators on CPUs, one developer moved the Rocket League physics engine onto the GPU, running thousands of games in parallel — training an agent that surpasses Nexto in just a few hours, when it would otherwise take weeks.

From PureJaxRL to CARL: A Two-Year Engineering Adventure

The project was inspired by PureJaxRL — an open-source project that implements fully parallelized RL environments in JAX. Motivated by this, the developer decided to build a similar GPU-parallel environment for Rocket League.

The existing RocketSim is a high-performance CPU simulator that has been integrated into RLGym for RL training. But CPU parallelism is limited, and when training large-scale neural networks, data transfer between host and device becomes a bottleneck. So the developer spent nearly two years (on and off) porting the physics logic to CUDA, naming the project CARL — CUDA-Accelerated Rocket League.

The core idea behind the project isn't complicated, but it's extremely challenging to implement: keep all simulation state in GPU memory and completely eliminate host-device data transfer overhead during training.

JAX and PureJaxRL: JAX is a high-performance numerical computing library developed by Google, supporting automatic differentiation, vectorization, and just-in-time compilation (JIT), and can compile computation graphs to GPU/TPU via XLA. PureJaxRL is a JAX-based reinforcement learning library that uses primitives like jax.vmap and jax.lax.scan to express the entire RL training loop (including environment simulation, policy inference, and parameter updates) as pure functions, enabling parallel execution of thousands of environment instances on the GPU. Unlike traditional CPU environments (such as Gym), PureJaxRL's environment state is also JAX arrays, so it can naturally reside on the device. CARL borrows this design, but for Rocket League's complex rigid-body physics and collision detection, it hand-writes customized simulation kernels in CUDA to efficiently advance physics state on the GPU.

Device-Resident Simulation: Why Is It So Critical?

In traditional RL training pipelines, environment simulation runs on the CPU while neural network inference runs on the GPU. Each step requires transferring observations from CPU to GPU, then sending action results back to CPU — this back-and-forth copying seems trivial, but over millions of training steps it accumulates into enormous time overhead.

CARL's approach is to keep the entire simulation loop on the GPU. The states of thousands of games reside in GPU memory simultaneously, with the GPU's thousands of cores advancing each game's physics simulation in parallel. This not only eliminates data transfer but also fully leverages the GPU's throughput.

This philosophy is in line with PureJaxRL, but CARL is specifically optimized for Rocket League's complex physics. The developer mentions that he learned a great deal about GPU programming and performance optimization along the way, while also admitting there's "still a lot to learn."

Host-Device Data Transfer Bottleneck: In a typical RL training loop, environment simulation (such as game physics stepping) executes on the CPU, while the policy network's forward/backward propagation completes on the GPU. Each step requires copying observations (state) from host memory to device memory, then copying actions from device memory back to host memory, after which the CPU can advance the simulation. Although the latency of each individual copy is not high, training typically requires millions to billions of steps, and the accumulated copy time far exceeds the actual computation time. Moreover, PCIe bus bandwidth is limited, and frequent small-batch transfers cannot fully utilize bandwidth, further amplifying overhead. The core idea of device-resident simulation is to keep both simulation state and computation graphs permanently in GPU memory, so that the entire "simulation→inference→action→simulation" loop completes entirely within the device, synchronizing only through kernel launches, thereby eliminating copy overhead and leveraging the GPU's thousands of cores to accelerate physics stepping in parallel.

Training Speed: From Weeks to Hours

The most impressive result comes from a comparative experiment. The developer used the default training configuration of the well-known Rocket League AI bot Nexto to train an agent, and within just a few hours it reached Nexto's level.

Nexto's original training, according to the developer, took weeks of wall clock time.

This comparison is not a strict controlled experiment — hardware configurations, implementation details, etc. may differ — but the order-of-magnitude difference is enough to demonstrate the potential of device-resident simulation. More importantly, the trained agent can seamlessly transfer to the real game, indicating that CARL's physics simulation maintains good consistency with the actual game.

Nexto and Rocket League Agent Skill Levels: Nexto is a famous reinforcement learning-based bot in the Rocket League community, trained by developer Roland, whose skill level has reached or even surpassed top human players (such as Grand Champion rank). Nexto was trained on CPU using RLGym and RocketSim, at a massive scale that took weeks of real time. CARL uses the same training configuration as Nexto (including reward functions, hyperparameters, and network architecture), but with GPU-parallel simulation, it reaches a comparable competitive level within hours. It should be noted that this comparison is not a strict controlled experiment, since the hardware differs and the simulator implementations also vary, but the training time reduction from weeks to hours — an order-of-magnitude improvement — is sufficient to prove the tremendous potential of device-resident simulation.

Feature-Complete, But Still Room for Improvement

The developer states that CARL has now reached "feature-complete" status, but several items remain to be addressed:

• Built-in training pack API: for targeted training of specific skills (such as aerial dribbling, defense, etc.)
• Speed and consistency benchmarks: systematically evaluating CARL's physics deviation from RocketSim
• Testing and engineering best practices: improving code quality and maintainability
• JAX and TensorFlow integration: exploring compatibility with mainstream RL frameworks

The developer plans to release demo videos and more detailed performance comparisons in the future.

Community Response and Significance

After being posted on Reddit, the project attracted attention. For RL researchers, CARL demonstrates a clear path: migrating environment simulation from CPU to GPU can bring order-of-magnitude training speedups. This approach not only applies to Rocket League but can also be extended to other RL tasks requiring complex physics simulation.

Of course, fully implementing a complex physics engine on the GPU is no easy feat. The developer spent two years to get to this point, and physics consistency validation is still ongoing. But the direction is clear — when simulation becomes the bottleneck, running the simulation itself on the GPU may be the most direct solution.

The developer says the project code and more details will be released gradually after further refinement.

One-line Summary

The developer ported Rocket League's physics engine to the GPU with CUDA, training an agent to Nexto's level in just hours.

Paragraph Summary

This article introduces CARL, a project where the developer, inspired by PureJaxRL, spent two years migrating Rocket League's physics simulation from CPU to GPU, achieving device-resident simulation. In traditional RL training, environment simulation runs on CPU while network inference runs on GPU, with frequent data copying becoming a bottleneck. CARL keeps all simulation state resident in GPU memory, running thousands of games in parallel, completely eliminating host-device transfer overhead. Comparative experiments show that using Nexto's default training configuration, CARL reaches Nexto's level within hours, whereas the original training took weeks. The trained agent can seamlessly transfer to the real game. The project is now feature-complete, with future work on training pack APIs, benchmarks, and JAX/TF integration. This approach offers a viable path to order-of-magnitude acceleration for RL tasks requiring complex physics simulation.