Migrating from PyBullet to Isaac Sim: A Hands-On Guide to Reinforcement Learning with Custom Robots

From PyBullet to Isaac Sim: A Real-World Migration Story

For many robotics enthusiasts and researchers, PyBullet is often the first stop in physics simulation — it's lightweight, easy to pick up, and well-documented. PyBullet is a Python wrapper around the Bullet Physics engine, developed by Erwin Coumans and extensively maintained during his time at Google Brain. Bullet Physics was originally designed for games and visual effects in film (it was used in AAA titles like Red Dead Redemption 2) before being widely adopted in the robotics simulation community. Its core strengths lie in pure CPU computation, cross-platform compatibility, and a clean API — just a few lines of code can load a URDF robot model and start running simulations. However, as training tasks grow in complexity and scale, PyBullet's CPU-based serial simulation quickly becomes a bottleneck.

NVIDIA's Isaac Sim, with its GPU-accelerated physics engine, has emerged as the natural next step. Isaac Sim is part of the NVIDIA Omniverse platform, built on top of the PhysX 5 physics engine, which achieves native GPU parallelization of physics computations via CUDA. Isaac Sim also integrates an RTX renderer with ray-tracing-level visual fidelity — particularly important for robot policy training that relies on visual input. That said, its steep learning curve has discouraged more than a few developers.

Recently, a Reddit developer shared their complete experience migrating from PyBullet to Isaac Sim, successfully building their first reinforcement learning training script for a custom robot. The value of this case study isn't in showing off — it's in honestly documenting how an individual developer tackled Isaac Sim with limited hardware (a laptop equipped with a mobile RTX 3070 GPU).

Where Exactly Is Isaac Sim's Entry Barrier?

The author admitted that Isaac Sim's complexity had kept him from trying it for a long time. He had previously completed various robotics tasks in PyBullet and only recently took the plunge. Just getting the simulator to run in windowed mode on a laptop — so he could visually inspect the training process and catch early issues — took a significant amount of time debugging various settings.

This serves as a reminder for newcomers: the main cost of getting started with Isaac Sim lies in environment setup and parameter tuning, not in the algorithms themselves. Specifically, Isaac Sim depends on the full Omniverse platform installation chain (including a Nucleus server, specific CUDA versions, and driver compatibility). Robot models are recommended to use the USD (Universal Scene Description) format — an open scene description standard developed by Pixar and adopted by NVIDIA's Omniverse platform, supporting richer materials, physics properties, and hierarchical composition. This introduces some conversion and adaptation overhead compared to the URDF format commonly used in the ROS ecosystem.

PPO Control Strategy Design: Controlling the "Controller," Not Joint Angles

The most interesting design decision in this project is the choice of action space for reinforcement learning. Rather than having PPO (Proximal Policy Optimization) directly output joint angles, the author handed the robot's directional controller inputs to the agent.

PPO is a policy gradient reinforcement learning algorithm proposed by OpenAI in 2017, developed by John Schulman and colleagues. It addressed the high computational complexity of its predecessor TRPO (Trust Region Policy Optimization) by introducing a clipping mechanism that limits the magnitude of policy updates, ensuring each iteration doesn't stray too far from the current policy. The core idea behind PPO is to maximize expected returns while constraining the KL divergence between old and new policies through a clipped ratio term, achieving a balance between training stability and sample efficiency. Thanks to its simple implementation, low sensitivity to hyperparameters, and broad applicability, PPO has become the de facto baseline algorithm in robot reinforcement learning.

Why This Action Space Design Matters

In other words, PPO has to learn to climb slopes by issuing directional controller commands — much like a human operating a remote control — combined with body posture adjustments like pitch and height. This approach is fundamentally different from direct joint angle control:

• Closer to real deployment scenarios: Real robots are typically driven through high-level controller interfaces, not by precisely controlling individual joints. Having the policy learn controller-level commands means the trained model transfers more easily to a physical robot.
• Reduced action space dimensionality: Directly controlling all joint angles of a multi-legged robot creates a massive action space. Abstracting through a directional controller optimizes both learning difficulty and convergence speed.
• Preserves low-level controller stability: The robot's own controller handles translating high-level commands into specific joint motions, while the policy focuses solely on the decision layer — "where to go and how to adjust posture."

This "hierarchical control" approach is extremely common in real robot deployments. Hierarchical Control is a classic architectural pattern in robotics, typically divided into task planning, motion planning, and low-level execution layers. In quadruped or hexapod robots, low-level controllers (such as Central Pattern Generators (CPG) or Inverse Kinematics (IK) solvers) translate high-level velocity and direction commands into specific joint trajectories, ensuring gait stability and smoothness. The engineering advantage of this layered design is decoupling — the high-level policy doesn't need to understand the dynamics of each joint motor; it only needs to output semantically clear control commands. In sim-to-real transfer, this architecture also significantly reduces the impact of the reality gap, since the low-level controller can be independently tuned on the physical robot while the high-level policy remains unchanged.

GPU-Accelerated Simulation Performance: Isaac Sim vs. PyBullet

The performance data is the most tangible takeaway from this experiment. For the same reinforcement learning training task:

Simulation Platform	Hardware	Training Time
Isaac Sim	RTX 3070 Mobile GPU	~45 minutes
PyBullet	CPU	~1 hour 45 minutes

On laptop hardware, GPU-accelerated simulation cut training time to roughly 43% of the original — more than a 2x speedup. Considering this was achieved on a mobile RTX 3070, switching to a desktop or data-center-grade GPU with thousands of parallel environment instances would yield even more dramatic acceleration.

Why GPU Parallel Simulation Delivers Such Massive Speedups

Isaac Sim's core advantage is that its physics simulation runs directly on the GPU, enabling massive parallelization of multiple simulation environments. This GPU parallel simulation capability is primarily delivered through the Isaac Lab framework (formerly known as Isaac Gym). The key technology maps all rigid body dynamics, collision detection, and constraint solving onto GPU CUDA cores. In traditional CPU simulation, each environment runs as a separate serial process; in the GPU parallel architecture, the states of thousands of simulation environments are stored as tensors in GPU memory, and physics steps are completed simultaneously through batched matrix operations.

This design has another critical advantage: simulation state data stays resident in GPU memory and can be passed with zero-copy overhead to neural networks running on the same GPU for policy inference and gradient computation, completely eliminating the CPU-GPU data transfer bottleneck. According to NVIDIA's official data, high-end GPUs can run over 4,096 parallel environments simultaneously, achieving data collection rates tens or even hundreds of times faster than CPU simulation.

Reinforcement learning is fundamentally data-intensive — agents need massive amounts of interaction samples to learn policies. When simulation itself can collect data in parallel, sample collection speed increases by orders of magnitude. This is Isaac Sim's greatest value proposition over CPU-based simulation. For individual developers with limited computational resources, it means completing training tasks that were previously prohibitively time-consuming within reasonable timeframes.

Next Steps: Whole-Body Locomotion and Physical Deployment

The author is currently working toward a more ambitious goal — complete full locomotion simulation, again using his custom robot. He admits it's considerably more complex than expected, but the end goal is clear: train a complete locomotion control model and load it onto a real 3D-printed robot.

Sim-to-Real: Closing the Loop from Simulation to Physical Hardware

This goal outlines a complete development pipeline:

1. Build a custom robot model in Isaac Sim
2. Train locomotion control policies through reinforcement learning
3. Deploy the trained model to a physical robot

From controller-based slope climbing to full-body locomotion policy learning to physical deployment — this is exactly the prevailing research paradigm in robot learning today.

The fundamental challenge of sim-to-real transfer is the "Reality Gap" — simulation environments cannot perfectly replicate real-world physics, including non-uniform friction coefficients, motor response delays, sensor noise, and ground irregularities. Current mainstream approaches include: Domain Randomization, which randomly perturbs simulation parameters (such as mass, friction, observation noise) during training to force the policy to learn robustness to uncertainty; System Identification, which narrows the simulation gap by precisely measuring the physical parameters of the real robot; and progressive transfer, which pre-trains in simulation and then fine-tunes with a small amount of real-world data. In recent years, ETH Zurich's ANYmal quadruped and multiple research projects from UC Berkeley have demonstrated that carefully designed simulation training strategies can be deployed to physical robots with near-zero adjustment.

The author's choice to have the policy control high-level controller inputs rather than joint angles also paves the way for subsequent sim-to-real transfer — because the physical robot is driven through exactly the same controller interface.

Open-Source Resources: Code, Videos, and Hardware Info All Included

Commendably, the author didn't just share results — he open-sourced the entire process. He published a companion tutorial video on YouTube and uploaded the simulation scripts to a GitHub repository (HexaDogZBD-IsaacSim-RL). He also provides introductory videos about the real 3D-printed robot itself.

For developers looking to get started with reinforcement learning in Isaac Sim, a resource package containing complete code, video walkthroughs, and hardware specifications is far more valuable than a simple results showcase. It demonstrates that even on consumer-grade laptop hardware, individual developers can complete the full robot learning pipeline from simulation training to physical deployment.

Conclusion

Although modest in scale, this case study precisely highlights several key elements of modern robot learning: the efficiency revolution brought by GPU-accelerated simulation, deployment-oriented action space design, and a complete closed-loop approach from simulation to reality. For developers who have been deterred by Isaac Sim's complexity, a real, reproducible hands-on experience like this might be exactly the confidence boost needed to take that first step.