Porting NES to CUDA: 20x Speedup in Mario Reinforcement Learning Training

When the Bottleneck Isn't Learning—It's the Simulator

In reinforcement learning (RL) practice, we typically focus on policy network design, hyperparameter tuning, or algorithm convergence. But for game-based environments, a commonly overlooked yet critical bottleneck is often the environment simulator's throughput itself.

A developer recently shared his experience on Reddit. While learning the PPO (Proximal Policy Optimization) algorithm, he chose the classic Super Mario Bros. as his project vehicle. PPO is a policy gradient algorithm introduced by OpenAI in 2017 that has become one of the most widely used algorithms in reinforcement learning due to its simple implementation, ease of tuning, and robust performance. PPO's core idea is to introduce a clipping mechanism during policy updates, limiting the divergence between new and old policies to ensure training stability while achieving efficient policy improvement. A typical PPO training loop consists of two alternating phases: first, the "rollout phase," where the agent executes actions in the environment according to the current policy and collects state-action-reward trajectory data; then the "learning phase," where the collected trajectory data is used to update the policy and value networks. The time ratio between these two phases directly determines overall training efficiency—if the rollout phase becomes a bottleneck due to simulator speed, even the best learning algorithm cannot perform effectively.

However, he quickly discovered that the speed limitation in each training run came not from the learner, but from the simulator running the NES game.

The specific numbers were striking: the commonly used Python NES simulator nes-py only achieves approximately 132 env-steps/s on a single CPU core. This means that a 25-million-step (25M-step) training run would require over two days just for "pressing buttons to advance the game frames"—and that's before counting the learning component.

Why Existing GPU Simulation Solutions Aren't Enough

Facing the simulator bottleneck, the industry actually has mature approaches—move the simulator to the GPU and run it in parallel. NVIDIA's CuLE (CUDA Learning Environment) is such a project, capable of running thousands of game environments in parallel on the GPU, keeping observation data in GPU memory to avoid expensive CPU-GPU data transfers.

CuLE is a research project released by NVIDIA in 2019, with its core contribution being the complete porting of the Atari 2600 simulator (based on Stella) to run on CUDA. The Atari 2600's hardware architecture is simpler than NES—with only 128 bytes of RAM and relatively simple graphics hardware (TIA chip), making GPU porting engineering more manageable. CuLE's key design philosophy is: simultaneously run thousands of independent game instances on the GPU, with each instance occupying one or a few GPU threads, keeping all environment states and observation data in GPU memory. This design eliminates the data transfer bottleneck between CPU and GPU in traditional approaches—PCIe bandwidth is typically 12-32 GB/s, while GPU memory bandwidth can reach hundreds of GB/s or even TB/s levels. CuLE demonstrated in their paper the ability to achieve throughput of hundreds of thousands of frames per second on a single V100, proving the transformative impact of GPU-native simulators on RL training efficiency.

The problem is that CuLE only supports Atari games and doesn't cover Super Mario Bros. on the NES platform. This left the developer with a choice: either abandon Mario for Atari games, or do it himself—port the entire NES simulator into CUDA kernels.

He chose the latter. This is precisely the origin of the NeSLE project.

Technical Core: Running an Entire NES Console in GPU Threads

The most hardcore aspect of this project is that it implements the core components of NES—the 6502 CPU, PPU (Picture Processing Unit), and bus—entirely as CUDA kernels, adopting a "one thread per environment" parallel mode.

The NES (Nintendo Entertainment System) is an 8-bit home console released by Nintendo in 1983, with its core architecture consisting of three main components: a custom CPU based on the MOS 6502 (Ricoh 2A03), responsible for running game logic and audio processing; a PPU (Picture Processing Unit), responsible for rendering background tiles, sprites, and generating video signals; and the bus system connecting them, responsible for address mapping and data transfer. An emulator must precisely replicate the behavior and timing relationships of these components at the instruction level—each instruction of the 6502 CPU consumes a specific number of clock cycles, the PPU and CPU run synchronously at a 3:1 clock ratio, and each frame requires precise simulation of tens of thousands of clock cycles. Porting this logic from traditional serial C/Python implementations to CUDA parallel kernels means each GPU thread must independently maintain a complete set of NES hardware states, including CPU registers, memory mapping, and PPU rendering state—a non-trivial engineering challenge.

Observation Data Never Leaves GPU Memory

In traditional RL training workflows, the simulator generates observations on the CPU, then copies them to the GPU for neural network inference, and the trained actions are copied back to the CPU for execution—this back-and-forth transfer overhead becomes a new bottleneck under high parallelism.

NeSLE's approach is to keep observation data permanently resident on the GPU. Furthermore, the PPO training loop directly reads rollout data from the GPU via DLPack, rather than using Stable-Baselines3's (SB3) default CPU rollout buffer.

DLPack is an open memory tensor structure standard designed to enable zero-copy tensor data sharing between different deep learning frameworks (such as PyTorch, TensorFlow, JAX, CuPy, etc.). Its core idea is to define a framework-agnostic C-level data structure (DLTensor) describing the tensor's data pointer, shape, stride, data type, and device information—any framework that can parse this structure can directly access the underlying data without copying. In NeSLE's scenario, observation data produced by CUDA kernels exists as raw pointers in GPU memory, and after being wrapped with DLPack, PyTorch can directly use it as tensors, completely avoiding unnecessary data transfers between CPU and GPU. This means the entire data pipeline from simulation, sampling to learning stays on the device as much as possible.

Seamless Compatibility with Stable-Baselines3 Ecosystem

A commendable engineering detail is that this implementation is packaged as an SB3-compatible VecEnv. Stable-Baselines3 (SB3) is currently one of the most mainstream reinforcement learning algorithm libraries in the Python community, providing high-quality PyTorch implementations of classic algorithms like PPO, A2C, SAC, and TD3. A core design of SB3 is the VecEnv (Vectorized Environment) abstraction layer—it wraps multiple environment instances into a unified interface, supporting batch step and reset operations. SB3 comes with implementations like SubprocVecEnv (parallelism via multiprocessing) and DummyVecEnv (serial simulation).

NeSLE packages itself as a custom VecEnv, meaning users only need to replace one line of environment creation code to switch the underlying implementation from CPU multiprocess simulation to GPU parallel simulation, while the upper-level PPO training code requires no changes. This compatibility with the existing ecosystem significantly lowers the barrier to adopting GPU simulators. Additionally, the project provides a "GPU-resident PPO" implementation, allowing the entire training loop to run in GPU memory.

Performance Data: What 20x Speedup Really Means

The core metrics provided by the author are very direct: on an entry-level GTX 1050 Ti, running 25 million steps with 2048 parallel environments, including the learning phase, wall-clock time is only 2.5 hours.

For comparison, the same task with single-process nes-py requires approximately 52 hours—a speedup ratio approaching 20x, achieved on a several-years-old low-end graphics card.

Distinguishing "Simulation Throughput" from "Training Throughput"

The author maintains admirable rigor here. He specifically notes that running the simulator purely on an A100 (with 65,536 environments) can achieve an astounding throughput of 3.27 million env-steps/s, but this is only simulator throughput, not training throughput.

In real training, the wall-clock time occupied by the simulator drops to less than 2%, with the remaining time mainly spent on policy network computation and rollout data management. In other words, once the simulator is moved to GPU, the bottleneck shifts from "running the game" to "training itself"—this is exactly the healthy state we want to see. This is also a vivid illustration of the classic Amdahl's law in system optimization: when one component is no longer the bottleneck, the system's overall performance ceiling is determined by the remaining slowest component.

Current Limitations and Considerations

The author is also candid about the project's limitations, and this honesty deserves to be faithfully presented in technical communication:

• The 52-hour baseline is single-process. A fully multi-core CPU solution would close a significant portion of the gap, but the author hasn't measured this number yet, so the 20x speedup would converge somewhat in multi-core comparisons.
• Performance breakdown hasn't been fully profiled on Linux. How much time policy computation and the rollout pipeline each occupy hasn't been precisely measured.
• Only supports Mapper 0. NES cartridges extend addressing capabilities through different "Mapper" chips; currently NeSLE only supports the most basic Mapper 0, limiting the range of games it can run. NES Mappers (memory mappers) are a key concept for understanding the diversity of its cartridge technology. The NES CPU has only a 16-bit address space (64KB), and the PPU's address space is also very limited. As game capacity and complexity grew, developers integrated additional chips on cartridges to implement bank switching, breaking through address space limitations. The iNES format defines over 250 different Mapper numbers, though only about twenty to thirty are commonly used. Mapper 0 (i.e., NROM) is the most basic mapping method, without any bank switching logic, supporting up to 32KB program ROM and 8KB graphics ROM. The classic Super Mario Bros. original happens to use Mapper 0, so NeSLE can run it; but Super Mario Bros. 3 uses Mapper 4 (MMC3), and more complex games use other Mapper types. Each Mapper has unique register mapping and switching logic, and implementing them in CUDA requires writing corresponding kernel code for each one, explaining why Mapper support expansion is a gradual engineering task.

Implications for Reinforcement Learning Engineering Practice

Though this personal project is small, it clearly reveals an important principle in reinforcement learning engineering: before staring at algorithm optimization, figure out where the system's real bottleneck is.

Often, what constrains experimental iteration speed isn't the model or algorithm, but environment interaction throughput. When the simulator becomes the bottleneck, GPU-izing it, eliminating data transfers, and keeping the entire pipeline on the device can bring order-of-magnitude efficiency improvements—even an old card like the GTX 1050 Ti can compress a two-day experiment into an afternoon.

This approach isn't unique to NeSLE. In recent years, more projects in the RL community have taken the "environment GPU-ization" route: Google DeepMind's Brax moved physics simulation to run on JAX, NVIDIA's Isaac Gym natively deploys robot simulation environments on GPU, and the Pgx project implements various board game environments in JAX. These projects collectively point to a trend—unifying the entire RL training pipeline (environment simulation + policy learning) on the same accelerator device, eliminating data transfer overhead from heterogeneous computing. NeSLE provides a concrete and elegant case for classic game simulators along this path.

For developers hoping to reproduce or research, the project is open-sourced on GitHub (hbofz/NeSLE), usable both as a plug-and-play environment for SB3 and providing a complete GPU-resident PPO implementation.

Key Takeaways