GzDRL: An Open-Source Framework for Running Reinforcement Learning Directly in Gazebo

The field of robot reinforcement learning (RL) has long faced an awkward contradiction: training requires high-throughput, reproducible simulation environments, while deployment relies on mature robotics ecosystems (such as Gazebo/ROS). To prioritize training speed, researchers are often forced to switch to simulators completely different from their deployment stack, dramatically increasing the sim-to-real transfer cost. Recently, an open-source framework called GzDRL attracted attention on Reddit for attempting to run reinforcement learning training directly inside Gazebo — fundamentally eliminating this divide.

Why Do Reinforcement Learning Inside Gazebo

GzDRL's core motivation is to preserve Gazebo's complete robotics simulation ecosystem while bypassing the ROS/transport layer bottlenecks common during training.

In traditional approaches, RL action and observation data must be transmitted over ROS topics, creating a serious performance bottleneck during high-frequency training. Every interaction step incurs serialization, transmission, and deserialization overhead. When you need tens of millions of interaction steps to train a usable policy, this overhead compounds enormously.

GzDRL's approach is to use an explicit "action → physics → observation" execution sequence, turning simulation stepping into a deterministic, controllable process rather than relying on asynchronous message passing. The direct benefits are twofold: a significant boost in training speed, and strict reproducibility — the latter being one of the most criticized pain points in current RL research.

Technical Architecture: A High-Performance Backend Borrowed from EnvPool

GzDRL's performance foundation comes from a C++ backend that, according to the author, derives from EnvPool — a vectorized environment library renowned for its extremely high throughput.

Batched Parallelism and GIL Release

The framework supports batched parallel environments, meaning multiple Gazebo simulation instances run simultaneously to collect training experience. More critically, it allows environments to be created directly in Python while releasing the GIL (Global Interpreter Lock) during execution.

This is especially important for Python users. The GIL prevents pure Python from truly leveraging multi-core parallelism, but GzDRL's design — running simulation at the C++ layer and releasing the GIL — strikes a balance between "keeping the Python interface clean" and "achieving efficient vectorization." Developers can write environment logic in familiar Python while enjoying parallel performance close to native C++.

GIL (Global Interpreter Lock) is a mutex in the CPython interpreter that ensures only one thread executes Python bytecode at a time. While this simplifies memory management, it prevents multi-threaded Python programs from achieving true parallelism on CPU-intensive tasks — a common performance bottleneck in scientific computing and reinforcement learning. Libraries like NumPy and PyTorch handle this by actively releasing the GIL when calling underlying C/Fortran computations, allowing other Python threads to run concurrently. GzDRL follows the same strategy: simulation stepping executes at the C++ layer with the GIL released, so policy network inference and data processing on the Python side can proceed concurrently without blocking each other.

Features Designed for Different User Groups

The author designed corresponding features for three types of users:

• Researchers: Reproducible state transitions and training, batched vectorization, multi-agent environments, domain randomization, and multi-level control abstractions ranging from rotor commands to high-level references.
• Industry: Trained policies can remain entirely within the Gazebo/ROS workflow — no need to maintain two separate simulator stacks for training and deployment.
• Hobbyists and students: Supports CPU-only training, making RL experiments accessible without expensive high-end NVIDIA GPUs.

EnvPool is an open-source vectorized environment library developed by SeaAI Lab, designed for large-scale RL training. Its core idea is to implement environment logic in C++, manage multiple parallel environment instances via a thread pool, and overlap the "action execution" and "observation collection" phases in an asynchronous pipeline to maximize CPU utilization. On benchmarks like Atari, EnvPool's throughput can be tens of times higher than pure Python gym implementations. GzDRL borrows its C++ backend architecture to bring this high-throughput design into full physics simulation scenarios — a significant engineering achievement in migrating game RL infrastructure to the robotics domain.

Benchmarks: Dual Highlights of Throughput and Reproducibility

The benchmark data GzDRL published is the most compelling part of this release.

On a workstation CPU, GzDRL achieves 78.6k environment steps/s throughput. For a robot RL framework based on full physics simulation (rather than a simplified physics engine), this is a remarkably impressive number — especially considering it doesn't rely on GPU acceleration.

The reproducibility results are even more striking: the author ran five independent PPO training runs with the same random seed, producing completely identical learning curves, with 100% matching policy checkpoint hashes.

This is almost a luxury in the RL world. Due to factors like parallel environment scheduling order, floating-point non-determinism, and asynchronous communication, the vast majority of RL frameworks struggle to achieve bit-level exact reproducibility. GzDRL achieves strict training repeatability through its explicit deterministic execution sequence — a practically valuable property for reproducing research papers, verifying experimental results, and debugging.

Sim-to-Real: Direct Deployment to a Physical Quadrotor

No matter how good the simulation, one core question must ultimately be answered: can the trained policy work on a real robot?

GzDRL's answer: the author deployed a trajectory-tracking policy trained in simulation directly to a physical quadrotor, without any fine-tuning.

Zero-shot sim-to-real transfer typically relies on two prerequisites: the simulation physics must be close enough to reality, and sufficient domain randomization must be applied during training to enhance policy robustness. GzDRL preserves Gazebo's mature physics simulation capabilities while also including built-in domain randomization — which may be the key to achieving seamless sim-to-real transfer. Of course, a single case study is not enough to demonstrate universality across complex tasks; further community validation is still needed.

Domain Randomization is one of the mainstream techniques for improving sim-to-real transfer success rates, systematically popularized by OpenAI in their robotic dexterous manipulation research. The principle is to randomly perturb physical parameters of the simulation during training — such as mass, friction coefficients, motor delays, sensor noise, etc. — forcing the policy to learn robust behaviors that work across a wide parameter distribution. When real-world parameters fall within the range covered by the randomization during training, the policy can transfer with minimal performance degradation. For dynamics-sensitive platforms like quadrotors, factors such as aerodynamic model errors and motor response delays can cause policy failure, making thorough domain randomization especially critical.

Observations and Outlook

GzDRL's value lies not in inventing new RL algorithms, but in its engineering integration philosophy: grafting EnvPool-style high-performance vectorization onto the de facto standard Gazebo/ROS ecosystem in robotics, while treating "reproducibility" as a first-class citizen.

For robot developers who have long been forced to trade off between training efficiency and ecosystem compatibility, this "no stack switching" approach is extremely appealing. The CPU-only training support also significantly lowers the barrier to entry, enabling students and hobbyists without high-end GPUs to participate in robot RL research.

The project is now open-source on GitHub (github.com/amaldevh/gz-drl), and the author has publicly invited feedback, feature requests, and suggestions for task types to support. As an emerging framework, its maturity, documentation quality, and performance across diverse tasks still need to be proven over time — but its design philosophy is undoubtedly worth continued attention from the robot RL community.