Practical RL Compute Optimization: Core Strategies to Boost CPU and GPU Utilization

The Overlooked Compute Utilization Problem in RL Projects

In the study and practice of Reinforcement Learning (RL), the vast majority of tutorials focus on the algorithms themselves—from DQN, PPO to SAC, developers spend enormous amounts of time understanding policy gradients, value functions, and reward design. However, an extremely critical yet frequently overlooked question in real-world engineering is: how do you fully utilize the compute resources at hand?

The importance of this question has long been validated in industry. From the TPU clusters DeepMind used to train AlphaGo Zero, to the tens of thousands of CPU cores OpenAI Five relied on for parallel sampling, every top-tier RL achievement is backed by a carefully designed compute scheduling system. In fact, RL Systems Engineering has gradually emerged as an independent research direction—how to maximize sample throughput within a given compute budget is just as important as the design of the algorithm itself. For researchers with limited resources, improving compute utilization can sometimes yield more significant training speedups than switching to a more advanced algorithm.

It's worth noting that different RL algorithms differ significantly in their consumption of compute resources. DQN (Deep Q-Network), proposed by DeepMind in 2013, was the first to combine deep neural networks with Q-learning. Its Experience Replay mechanism not only breaks the temporal correlation between samples but also lays the foundation for asynchronous sampling and training, giving it a natural decoupling advantage. PPO (Proximal Policy Optimization), proposed by OpenAI in 2017, is known for its simple implementation and robustness to hyperparameters, and is currently the most widely used on-policy algorithm. It requires that sampled data must come from the current policy, meaning old data becomes obsolete immediately after each network update—making its sampling efficiency extremely limited and its need for parallelization the most urgent. SAC (Soft Actor-Critic), proposed by the Berkeley team in 2018, introduces a maximum entropy framework. As an off-policy algorithm, it can reuse historical experience, relatively alleviating sampling pressure, while performing better on the exploration-exploitation balance. Understanding the design philosophy and computational characteristics of these algorithms is a prerequisite for choosing the right compute optimization strategy.

Recently, a developer on Reddit hit the pain point directly: "We always learn how to do RL, but never learn how to use compute optimally. Does anyone have relevant experience? Such as better parallelization, MPI, distributed RL, etc."

The reason this question matters is that RL and supervised learning differ fundamentally in their computational characteristics. Supervised learning is typically a GPU-intensive task—feeding batches of data to the model can max out the GPU. RL, however, involves alternating between two phases: environment interaction (CPU-intensive) and policy optimization (GPU-intensive). Without optimization, you often end up in the awkward situation where the GPU sits idle waiting for the CPU to sample, or the CPU is fully loaded while the GPU is idle.

Why Reinforcement Learning Is Especially Prone to Wasting Compute

The Inherent Bottleneck Between Sampling and Training

A typical RL training loop consists of two phases: Rollout (the agent interacts with the environment to collect experience) and Learning (using the collected data to update the network). The former relies on extensive environment simulation (such as physics engines and game simulators) and mainly consumes CPU; the latter is the forward and backward propagation of neural networks and mainly consumes GPU.

The key problem is that these two phases are often executed serially. When the agent is sampling in the environment, the GPU is idle; when the network is updating, the CPU has nothing to do. For environments like MuJoCo and Atari, the sampling speed of a single environment often cannot keep up with the GPU's computing power, causing GPU utilization to stay at 20%–40% or even lower for long periods.

Take PPO training on MuJoCo continuous control tasks as an example: the theoretical compute of an RTX 3090 can easily support the data stream produced by hundreds of parallel environment instances, but if you only run a single environment, actual GPU utilization is usually below 10%, with the vast majority of compute waiting for the CPU to simulate the next physics step. The root cause of this phenomenon is that MuJoCo's underlying physics solver (a serial solving algorithm based on constraint optimization) makes it inherently difficult to parallelize a single instance—horizontal scaling by increasing the number of environments is the only path to breaking through this bottleneck.

From an information-theoretic perspective, this inefficiency is essentially a kind of "waiting-time entropy"—the GPU has extremely high compute bandwidth but stays in a low-entropy (low-information-processing) state for long periods due to insufficient data supply. The standard metric for quantifying this waste is MFU (Model FLOP Utilization), i.e., the ratio of the effective floating-point operations actually completed to the hardware's peak compute. Large language model training can typically reach an MFU of 40%–60%, whereas naive RL training often has an MFU below 5%—a staggering gap.

How to calculate MFU: MFU = (actual floating-point operations per second) ÷ (hardware peak FLOPS). For Transformer-class networks, actual FLOPS can be precisely estimated from model parameter count, batch size, and sequence length. For RL scenarios, you also need to factor the CPU wait time of the sampling phase into the overall throughput calculation, which makes measuring MFU for RL more complex than for supervised learning—but also more truly reflective of system efficiency.

Small Batches, Frequent Updates

Unlike training large language models, which process massive amounts of tokens at a time, RL's updates use relatively small batches and update frequently. This means the GPU frequently switches between "computing" and "waiting for data," making it difficult to maintain high throughput. The design philosophy of Vectorized Environments stems precisely from the SIMD (Single Instruction, Multiple Data) parallel computing paradigm: the same policy network simultaneously drives multiple environment instances, executing action inference and experience collection in batches, consolidating scattered small batches into the large-batch computation that GPUs prefer—fundamentally improving this situation.

This involves a basic principle of GPU computing: a GPU consists of thousands of relatively simple CUDA cores, and its design philosophy is to achieve high throughput by massively parallelizing simple operations, rather than processing complex serial logic with a small number of complex cores like a CPU. This means GPU computing efficiency is positively correlated with batch size—the larger the batch, the busier each CUDA core, and the higher the overall GPU utilization. When the batch is too small, a large number of CUDA cores sit idle, waiting for memory read/write operations to complete (i.e., "memory-bound" rather than "compute-bound"), which is the deep physical reason for GPU inefficiency in RL's small-batch scenarios.

Core Strategies for Improving CPU and GPU Utilization

1. Environment Parallelization (Most Direct and Effective)

The simplest way to improve CPU utilization is to run multiple environment instances in parallel. Mainstream RL frameworks all provide Vectorized Environments support:

• Gymnasium's SyncVectorEnv and AsyncVectorEnv: the latter samples in parallel via multiple processes, fully utilizing multi-core CPUs.
• Stable-Baselines3's SubprocVecEnv: distributes multiple environments across different subprocesses, bypassing the limitation of Python's GIL.

Here it's worth specifically explaining the impact of Python's GIL (Global Interpreter Lock). The GIL is a mechanism in the CPython interpreter that allows only one thread to execute Python bytecode at a time. This means that even with multithreading, Python code cannot achieve true CPU parallelism. For compute-intensive tasks like RL environment sampling, multithreading is almost useless. The reason SubprocVecEnv can bypass this limitation is that it uses multiple processes rather than multiple threads—each subprocess has its own independent Python interpreter and GIL, so they can truly run in parallel. It's worth noting that Python 3.13 introduced an experimental "no-GIL mode" (PEP 703), which may fundamentally change this limitation in the future, making multithreaded RL environment parallelism possible while significantly reducing the serialization overhead of inter-process communication.

The cost of multi-process parallelism is inter-process communication (IPC) overhead. SubprocVecEnv passes observations and actions between the main process and subprocesses through operating system Pipes, and each communication involves pickle serialization/deserialization of Python objects. When the observation space is large (such as high-resolution image inputs) or environment steps execute extremely fast, this serialization overhead can become a new bottleneck. In such cases, consider using a Shared Memory approach—the multiprocessing.shared_memory introduced in Python 3.8 allows processes to directly share numpy arrays, avoiding serialization overhead and increasing IPC bandwidth by several times in image observation scenarios.

In practice, choosing the number of environments requires comprehensive consideration: the CPU core count determines the upper limit of multi-process parallelism, while environment complexity (single-step simulation time) determines whether the inter-process communication overhead is worth bearing. For physics simulation environments like MuJoCo, single-step time is on the order of milliseconds, and multi-process gains are significant; whereas for simple discrete control environments, single-step time is only on the order of microseconds, and the overhead of inter-process pipe communication may dominate—in which case using SyncVectorEnv or pure Python batching is actually more efficient. Empirically, the number of environments should match the CPU core count; too many will actually reduce efficiency due to process scheduling overhead.

By running dozens or even hundreds of environments simultaneously, you can continuously supply data to the GPU, significantly reducing GPU wait time.

2. Decoupling Sampling and Training: The Actor-Learner Architecture

A more advanced approach is the distributed Actor-Learner architecture, which is also the core idea of "distributed RL." Representative frameworks include:

• IMPALA: multiple Actors continuously sample on the CPU, while one or more Learners focus on training on the GPU, passing data asynchronously to achieve complete parallelism between sampling and training.
• Ape-X / R2D2: a distributed experience replay architecture that uses a large number of Actors to fill a shared Replay Buffer.
• Ray RLlib: the most engineering-complete distributed RL library, supporting seamless scaling from single-machine multi-core to multi-machine clusters, automatically managing resource allocation between Actors and Learners.

IMPALA (Importance Weighted Actor-Learner Architecture), released by DeepMind in 2018, is a milestone in the field of large-scale distributed RL. Because the Actors use older versions of policy parameters during asynchronous sampling (the Learner is continuously updating the network, while the Actors' parameter updates lag behind), the data collected by IMPALA is technically "mildly off-policy" data—i.e., there is a slight difference between the data-generating policy and the current learning policy. To address this, IMPALA introduced the V-trace correction algorithm—its mathematical form is essentially a truncated importance sampling estimator. By introducing two truncation thresholds ρ̄ and c̄, it corrects policy bias while controlling variance, and can theoretically be proven to converge to the value function of a target policy implicitly defined by the truncation coefficients. This design embodies the exquisite combination of distributed systems engineering and RL theory: trading a small amount of algorithmic complexity for a substantial improvement in sampling efficiency. In Atari game tests, IMPALA achieves training speedups of tens of times compared to single-machine A3C, and its excellent performance on the DMLab-30 and Atari-57 multi-task benchmarks proves the dual advantages of asynchronous distributed architectures in both sample complexity and wall-clock time.

From a systems design perspective, the Actor-Learner architecture is essentially a specialization of the Producer-Consumer Pattern: the Actors, as producers, continuously write trajectory data to a shared Experience Buffer, while the Learner, as the consumer, draws batches from it for gradient updates. The buffer's capacity design is crucial—too small will cause the Learner to frequently block and wait, while too large may introduce severe Policy Lag, making the off-policy bias exceed the correction capacity of algorithms like V-trace. In engineering practice, the buffer size is usually set to 5–20 times the Learner's single consumption amount, striking a balance between traffic smoothing and data freshness.

Resource allocation strategy between Actors and Learners: In a single-machine multi-GPU environment, a common configuration is to allocate all CPU cores to the Actors (each Actor exclusively using 1-2 cores) and all GPUs to the Learner. Ray RLlib achieves zero-copy data sharing between Actors and Learners through its distributed object store (Plasma Store), further reducing communication overhead. In a multi-machine environment, the Actor nodes and Learner nodes are usually physically separated and connected via a high-speed network; the Actor nodes can be inexpensive CPU-only instances, significantly reducing overall cost.

The core goal of such architectures is: to ensure the GPU never runs short of data—while the Learner is training, the Actors have already prepared the next batch of experience in the background.

3. GPU-Side Environment Simulation

In recent years, a revolutionary direction has emerged—putting the environment simulation itself on the GPU as well. The sampling speed of traditional CPU environments is the biggest bottleneck, whereas GPU-accelerated simulators can run tens of thousands of environments in parallel:

• NVIDIA Isaac Gym / Isaac Lab: simulates thousands of robot environments in parallel on a single GPU, boosting sampling throughput by several orders of magnitude. Isaac Gym uses Position-Based Dynamics (PBD) and a TGS (Temporal Gauss-Seidel) solver, sacrificing some physical accuracy in exchange for the feasibility of large-scale parallelism—this is usually an acceptable trade-off for RL policy training, because what the agent needs is a sufficiently realistic physical response, not scientific-computing-level precision. Policy network inference, physics state updates, and gradient computation all happen on the same device, completely eliminating the bandwidth bottleneck of CPU-GPU data transfer. It's worth mentioning that GPU simulation also brings new challenges: the random number generation behavior on the GPU (CURAND) differs from the CPU version, and a large number of parallel environments under the same initialization conditions may produce patterned random correlations, requiring carefully designed seeding strategies to ensure exploration diversity.
• Brax: a differentiable physics engine based on JAX, running entirely on GPU/TPU. JAX's just-in-time compilation (JIT) and automatic vectorization (vmap) primitives make running thousands of parallel environment instances on a TPU an engineering task requiring only a few lines of code. Its "differentiable" nature allows directly taking gradients of the environment dynamics, opening new possibilities for model-based RL methods. Brax's differentiability has also given rise to the Analytic Policy Gradient method, i.e., backpropagating gradients directly through the Jacobian matrix of the environment dynamics, which can converge an order of magnitude faster than Monte Carlo policy gradient estimation on certain low-dimensional continuous control tasks.
• EnvPool: a high-performance environment pool implemented in C++ that dramatically improves sampling speed. EnvPool's core design pushes the environment's stepping logic down to the C++ layer, completely bypassing the performance bottleneck of the Python interpreter; it also uses a Thread Pool rather than a multi-process architecture, avoiding serialization overhead, and can achieve 5-20x throughput improvement over SubprocVecEnv on the Atari and MuJoCo benchmarks. EnvPool also natively supports an asynchronous execution mode, allowing it to continue processing environments that have completed their step while waiting for some environments to finish their current step, further filling in idle time on the CPU side.

With such approaches, the data transfer bottleneck between CPU and GPU is completely eliminated, and GPU utilization can approach 100%.

Distributed and Multi-Machine Scaling

MPI and Multi-Process Communication

For scenarios that need to scale across multiple machines, MPI (Message Passing Interface) is the classic choice. MPI is a cross-language standard for parallel computing communication, born in the high-performance computing field in the 1990s, allowing processes running on different nodes to collaborate through message passing. In the RL context, MPI is typically used to synchronize the gradients of multiple workers—each worker independently samples in its local environment and computes gradients, then aggregates all workers' gradients through MPI's AllReduce operation, effectively enlarging the overall batch size. OpenAI's early Baselines library made extensive use of MPI to implement multi-core parallel PPO.

The underlying implementation of the AllReduce operation usually uses the Ring-AllReduce algorithm—arranging the participating nodes into a logical ring topology, with gradient data passed and accumulated in segments along the ring path. The advantage of this algorithm is that the communication volume is independent of the number of nodes (the total amount of data each node sends and receives is fixed at 2*(N-1)/N gradient copies, where N is the number of nodes), making multi-machine scaling communication highly efficient. NCCL (NVIDIA Collective Communications Library) is a highly optimized implementation of Ring-AllReduce for GPU scenarios, and can approach the theoretical hardware bandwidth limit on multi-GPU systems connected by NVLink.

However, MPI requires all processes to be strictly synchronized. Once a worker is slower (the "Straggler Problem"), overall efficiency is dragged down—the fastest process must wait at the AllReduce barrier for the slowest process to finish before it can continue. In addition, MPI's deployment and configuration is relatively cumbersome, requiring manual management of process topology and communication primitives, with a high development barrier. Nowadays, more projects turn to higher-level abstraction frameworks like Ray, which internally encapsulate complex distributed communication details while supporting asynchronous scheduling, naturally avoiding the straggler problem.

From the network communication level, multi-machine RL training has very different bandwidth and latency requirements than large model training. Large model training, due to the enormous gradient tensors (often tens of GB), is extremely sensitive to inter-node bandwidth (InfiniBand, RoCE, etc.); whereas RL's distributed communication mainly transmits trajectory data (sequences of states, actions, and rewards)—each trajectory is relatively small in size but extremely high in frequency, making it more sensitive to latency. This means that in multi-machine RL cluster design, low-latency Ethernet (such as 25GbE) combined with efficient serialization protocols (Protocol Buffers, MessagePack) often improves actual throughput more than blindly pursuing high bandwidth.

Mixed Precision and Batching Optimization

Beyond architecture-level optimizations, there are also some general efficiency techniques:

• Mixed Precision Training (AMP): using FP16/BF16 to reduce memory footprint and accelerate computation. Modern GPUs (such as NVIDIA Ampere architecture and above) are equipped with dedicated Tensor Core units that provide hardware acceleration for FP16/BF16 matrix operations, theoretically boosting training throughput to 2-3x that of FP32 while halving memory footprint, allowing larger batches. It's worth noting that FP16 has only about 3.3 significant decimal digits and is prone to underflow during gradient computation; PyTorch's AMP solves this problem through Dynamic Loss Scaling. BF16 retains the same exponent width as FP32 and is often more stable than FP16 in RL training, requiring no loss scaling mechanism—making it the recommended choice on current A100/H100 GPUs. The usual practice is to store a master copy of parameters in FP32 and only convert to low-precision format during forward and backward propagation, ensuring the numerical stability of the optimizer state. It's especially important to note that the value function (Critic) output in RL may span multiple orders of magnitude (especially in sparse-reward or long-horizon tasks), and low-precision representation is more likely to cause numerical catastrophes; in practice, you can normalize the rewards or value targets (such as the PopArt algorithm) to enjoy mixed-precision acceleration while maintaining training stability.
• Increasing batch size: increase the batch size as much as memory allows to improve the GPU's per-computation efficiency.
• Asynchronous data prefetching: use background threads to move data from CPU to GPU in advance, hiding transfer latency.

Gradient Accumulation: when the required batch size for a single step exceeds the memory limit, you can split the large batch into several small batches, compute the forward pass and accumulate gradients for each small batch separately, and only perform a unified parameter update after all small batches have been processed. This technique allows researchers to still simulate the effect of large-batch training under limited single-GPU resources, and is an often-underestimated cost-effective optimization technique in RL engineering practice.

Practical Advice: Start With Performance Diagnosis

Before embarking on optimization, be sure to locate the bottleneck first. Use nvidia-smi or nvtop to observe GPU utilization, and use htop to observe CPU usage:

• Low GPU utilization while CPU is fully loaded → sampling bottleneck; you should prioritize increasing environment parallelism or adopting GPU simulation.
• CPU idle while GPU utilization is also low → a data transfer or batch-too-small problem.

More refined diagnosis can leverage PyTorch Profiler or NVIDIA Nsight Systems. These tools can visualize the working state of the CPU and GPU as a timeline, precisely locating the code paths corresponding to idle intervals—far more instructive than merely observing average utilization. PyTorch Profiler can also identify the launch overhead of CUDA Kernels, helping to discover the problem of an "excessively high proportion of launch latency" caused by batches that are too small, which is especially common in RL small-batch scenarios. Furthermore, Steps Per Second (SPS) is a more direct business metric than GPU utilization in RL performance optimization—it comprehensively reflects sampling efficiency, policy inference speed, and training step frequency, and is the "North Star metric" for measuring overall system efficiency. During optimization, you should treat improving SPS as the main optimization goal; improving GPU utilization is merely a means to achieve this goal, not the goal itself.

For most individual researchers and small teams, the following pragmatic path is recommended:

1. First use vectorized environments to max out the CPU cores;
2. Introduce Ray RLlib to decouple sampling and training;
3. If sampling is still the bottleneck and the budget allows, migrate to GPU-native simulation solutions like Isaac Gym or Brax.

Conclusion

RL compute optimization is a field severely underestimated by the teaching system. Algorithms are certainly important, but in real-world projects, whether training can be completed within a reasonable time, and whether expensive GPU resources can be pushed to their limits, often directly determines a project's success or failure. From environment parallelization to distributed Actor-Learner architectures, and then to GPU-native simulation, the core logic of this optimization path remains consistent throughout: eliminate waiting, and let the CPU and GPU each do their job, never sitting idle. Behind every layer of optimization lies its theoretical basis and engineering trade-offs—only by understanding these trade-offs can you make the technical choices best suited to your own resource constraints in a specific project, rather than blindly pursuing architectural complexity.

Key Takeaways