Reinforcement Learning for Obstacle Avoidance in CARLA: A Guide to Choosing Between DQN, PPO, and SAC

Introduction: The Reinforcement Learning Challenge in Autonomous Driving Obstacle Avoidance

In autonomous driving research, dynamic obstacle avoidance remains one of the most challenging core tasks. Recently, an undergraduate student shared their research project on Reddit: using reinforcement learning (RL) to achieve dynamic obstacle avoidance in the CARLA simulator. Their technical approach is quite representative—using LiDAR observation data, a discrete action space, and the DQN algorithm provided by Stable-Baselines3.

Behind this seemingly simple problem lie several fundamental decisions in applying reinforcement learning to robotics and autonomous driving: algorithm selection, reward design, and simulation environment trade-offs. This article provides an in-depth analysis of these three key issues, offering guidance for developers working on similar research.

Technical Setup for Obstacle Avoidance in CARLA: A Typical RL Environment Configuration

Before diving into algorithm selection, let's review the researcher's technical stack:

• Simulation Environment: CARLA simulator
• Observation Input: LiDAR raw 3D point cloud, reduced to a 1D array of 360 minimum distance values
• Algorithm: Stable-Baselines3's DQN
• Action Space: Discretized into 36 steering/throttle combinations
• Environment Wrapper: Custom Gym environment
• Objective: Avoid dynamic obstacles while continuously moving forward

The core characteristic of this design is compressing continuous sensor data (point cloud) into a structured 360-dimensional distance vector, while discretizing the inherently continuous vehicle controls (steering, throttle) into a finite set of action combinations. This simplification reduces engineering complexity but introduces trade-offs in algorithm selection.

Introduction to the CARLA Simulation Platform

CARLA (Car Learning to Act) is an open-source autonomous driving simulation platform jointly developed by Intel Labs, Toyota Research Institute, and the Computer Vision Center of Barcelona, first released in 2017. Built on Epic Games' Unreal Engine 4, it provides highly realistic urban environment rendering, including dynamic weather, lighting changes, pedestrian and vehicle traffic flows. CARLA supports multiple sensor simulations (RGB cameras, depth cameras, semantic segmentation cameras, LiDAR, GNSS, IMU, etc.) and provides a Python API for scenario orchestration and data collection. Due to its high fidelity and rich feature set, CARLA has become one of the de facto standard platforms for academic autonomous driving research, widely used in papers at top conferences such as CVPR, NeurIPS, and ICRA.

Stable-Baselines3: A Reliable RL Algorithm Library

Stable-Baselines3 (SB3) is a PyTorch-based reinforcement learning algorithm library, succeeding the earlier TensorFlow-based Stable-Baselines. It provides rigorously tested and benchmarked RL algorithm implementations, including DQN, PPO, SAC, A2C, TD3, and other mainstream algorithms. SB3's design philosophy is to provide reliable, reproducible implementations while maintaining code readability and extensibility. It's compatible with the OpenAI Gym (now Gymnasium) interface, allowing researchers to easily connect custom environments with standard algorithms, significantly lowering the barrier to RL application development.

LiDAR Data Dimensionality Reduction

LiDAR (Light Detection and Ranging) obtains precise 3D distance information about the surrounding environment by emitting laser pulses and measuring reflection time, generating raw data called point clouds. In autonomous driving, a single LiDAR point cloud frame may contain hundreds of thousands to millions of 3D coordinate points. Using this directly as RL input is too high-dimensional, increasing computational burden and making learning difficult. The researcher reduced it to minimum distance values across 360 directions, essentially projecting the 3D point cloud onto a 2D polar coordinate plane, similar to a 360-degree 2D laser scanner sweep. This processing preserves key obstacle bearing and distance information while compressing the state space from hundreds of thousands of dimensions to just 360, making it much easier for RL algorithms to extract useful features and learn effectively.

Comparing DQN, PPO, and SAC: How to Choose for Obstacle Avoidance Scenarios

Is DQN a Reasonable Baseline for Obstacle Avoidance?

The researcher's main reason for choosing DQN was that the action space is discrete—which is indeed DQN's natural domain. DQN (Deep Q-Network) makes decisions by learning Q-values for each discrete action, and for a scale of 36 action combinations, DQN is perfectly capable.

DQN was proposed by DeepMind in 2013 and published in Nature in 2015, representing a milestone in deep reinforcement learning. Its core idea is using deep neural networks to approximate the Q-function (state-action value function), solving the problem of traditional Q-learning being unable to store all state-action pairs in a table for high-dimensional state spaces. DQN introduced two key technical innovations: Experience Replay, which breaks temporal correlations between samples through random sampling of historical experiences, and Target Network, which stabilizes target values during training through delayed updates. As an off-policy algorithm, DQN can repeatedly leverage past collected experiences for learning.

As a baseline, DQN is a reasonable choice. It's simple to implement, relatively straightforward to debug, and has extensive mature experience for discrete action tasks. For an undergraduate research project, starting with DQN enables quickly establishing a complete working pipeline.

Advantages of PPO and SAC for Autonomous Driving Obstacle Avoidance

When we delve into the specific scenario of autonomous driving obstacle avoidance, the situation becomes more nuanced:

Natural fit for continuous actions: Vehicle steering and throttle are inherently continuous quantities. Discretizing them into 36 combinations means a loss of control precision—the vehicle cannot make fine adjustments between two discrete actions. SAC (Soft Actor-Critic), as a continuous action space algorithm, can output smooth, precise control signals, which may bring significant advantages in tasks requiring precise maneuvering like obstacle avoidance.

SAC was proposed by Tuomas Haarnoja et al. at UC Berkeley in 2018, an off-policy Actor-Critic algorithm based on the maximum entropy framework. Unlike traditional RL that only maximizes cumulative reward, SAC simultaneously maximizes policy entropy (randomness), meaning that given equal rewards, the algorithm prefers more exploratory policies. This design brings multiple benefits: better exploration capability avoids premature convergence to suboptimal solutions, stronger robustness against environmental perturbations, and lower sensitivity to hyperparameters reduces tuning difficulty. SAC uses Clipped Double-Q networks to reduce systematic overestimation bias in value functions and automatically adjusts the temperature coefficient α to dynamically balance reward maximization and entropy maximization.

SAC's sample efficiency advantage: SAC is an off-policy algorithm that maintains an experience replay buffer, allowing repeated use of historical data for multiple gradient updates. It typically has higher sample efficiency than the on-policy PPO (i.e., fewer environment interaction steps needed to achieve the same performance). Given that CARLA training is expensive—each simulation step requires physics simulation and potentially rendering computation—sample efficiency is a factor that cannot be ignored.

PPO's training stability: PPO (Proximal Policy Optimization) was proposed by John Schulman et al. at OpenAI in 2017. Its core idea is limiting the magnitude of policy updates through clipping the objective function, preventing training collapse from excessively large updates. While PPO's sample efficiency is lower than SAC's—since as an on-policy algorithm, old data must be discarded after each policy update—its training process is typically more stable and less sensitive to hyperparameters, making it a common choice in many autonomous driving RL research and industrial applications. OpenAI also chose PPO for the RLHF stage of training ChatGPT, precisely because of its reliable stability.

Practical Recommendations for Algorithm Selection

For projects like this, a pragmatic approach is: keep DQN as the discrete action baseline while introducing SAC with the action space converted back to continuous, and run comparative experiments. This both verifies whether continuous control actually brings performance improvements and makes the research report more compelling. Whether it's worth transforming the action space should ultimately be answered by experimental data, not a priori assumptions.

It's worth noting that Stable-Baselines3 provides out-of-the-box implementations for all three algorithms, and the code changes required to switch algorithms are minimal—the main work lies in redefining the action space (from Discrete to Box) and adjusting corresponding hyperparameters.

Reward Design in Reinforcement Learning: The Make-or-Break Factor for Obstacle Avoidance

Why Reward Engineering Determines Success or Failure

The researcher astutely recognized a core challenge: how to balance rewarding forward progress with penalizing collisions/unsafe behavior. This is precisely the most challenging part of reinforcement learning—reward engineering.

Reward engineering refers to the process of manually designing reward functions to guide agents toward learning desired behaviors. Since RL agents precisely maximize the given reward signal—this is the essence of their learning mechanism—any loopholes in reward function design can be "exploited." This is the "Reward Hacking" phenomenon, also known as the RL version of "Goodhart's Law": when a metric becomes a target, it ceases to be a good metric. This problem is also widely studied in AI Safety, as it reveals the fundamental difficulty of objective specification—there may be subtle but critical gaps between the behavior we actually want and the behavior we mathematically reward.

In obstacle avoidance tasks, a poorly designed reward function easily leads agents to find "shortcuts" for high returns without actually completing the intended task. The most typical failure mode is exactly what the researcher feared: the agent learns to simply stop to avoid collisions. If the collision penalty is too heavy and forward reward insufficient, stopping becomes a "local optimum"—no crashes (avoiding large negative rewards) and no risk-taking (since any movement carries collision risk).

Common Failure Modes in Obstacle Avoidance Reward Functions

Based on community experience, the following failure modes deserve attention:

1. Passive stopping: Excessive collision penalty causes the agent to prefer remaining stationary. The solution is introducing explicit "forward progress" rewards (e.g., based on longitudinal displacement along the planned path), or even applying a mild penalty for prolonged inactivity. The key is making the total return from "stopping" lower than from "cautious forward movement."

2. Aggressive rushing: Excessively high forward rewards cause the agent to ignore safety margins in pursuit of accumulated rewards, frequently resulting in close passes or collisions with obstacles. This can be mitigated by introducing continuous penalty terms related to obstacle distance (e.g., inverse proportional or exponential decay functions), making the agent feel continuously increasing "discomfort" when approaching obstacles.

3. Reward oscillation: Sparse rewards (only providing signals upon collision or goal reaching) make learning difficult, as the agent receives no meaningful feedback during most timesteps and cannot judge whether current behavior is good or bad. Dense rewards are recommended, providing per-step continuous feedback based on current speed, distance to obstacles, lane-keeping, heading angle deviation, etc. Dense rewards provide richer gradient signals, significantly accelerating learning convergence.

4. Circular behavior: If rewards are based solely on speed magnitude rather than actual forward direction, the agent may learn to spin in place to continuously receive speed rewards. The solution is using projected velocity relative to the target or path direction as the reward.

A relatively robust reward structure typically includes: positive speed/progress rewards, negative collision penalties (usually a large fixed value plus episode termination), gradual penalties based on safety distance (triggered when distance to the nearest obstacle falls below a threshold), and possibly smoothness rewards (penalizing sharp steering oscillations and acceleration changes, encouraging comfort). The key is keeping the magnitudes of each reward component balanced to prevent any single term from dominating the entire learning process. In practice, multiple experiments are usually needed to adjust the weight coefficients for each component, making this one of the most time-consuming aspects of RL applications.

Optimizing CARLA Simulator Training: Balancing Efficiency and Fidelity

CARLA's Advantages and Computational Cost

The researcher mentioned that while they enjoy using CARLA, it runs heavy, preventing sufficient training iterations. This captures CARLA's core contradiction.

CARLA is currently one of the most mainstream high-fidelity simulators in autonomous driving research, providing realistic urban scenes, complete sensor suites (cameras, LiDAR, radar), and rich traffic participants. For projects pursuing realism and research credibility, CARLA is an excellent choice. Its built-in Autopilot and ScenarioRunner tools also facilitate creating complex test scenarios.

However, the cost of high fidelity is heavy computational overhead. CARLA is based on Unreal Engine 4, and even at lower rendering quality settings, it demands significant GPU memory and compute power (typically requiring at least a dedicated GPU with 8GB VRAM). Simulation stepping is slow—under standard settings, CARLA's simulation rate is typically around 10-30 FPS, while RL training often requires millions or even tens of millions of environment interaction steps. Simple calculation: collecting 5 million steps at 20 FPS requires nearly 70 hours of pure simulation time, not including model training time. For RL training that depends on massive interaction sampling, this is a very real bottleneck.

Speeding Up CARLA Training and Alternative Simulator Options

To address training efficiency, consider the following strategies:

• Reduce CARLA rendering quality: Disable unnecessary rendering (if only using LiDAR without camera images, visual rendering can be skipped entirely), run in headless/off-screen mode, reduce world rendering resolution and detail levels—this can significantly speed things up, potentially achieving 2-5x speed improvements in some cases.
• Parallel environments: Run multiple CARLA instances simultaneously (each on different ports) for parallel sampling. Combined with SB3's SubprocVecEnv, this can linearly scale data collection rates, provided hardware resources (mainly GPU memory and CPU cores) are sufficient.
• Asynchronous mode and fixed time steps: CARLA supports synchronous mode, ensuring strict step alignment between simulation and client, avoiding time waste from waiting for rendering while ensuring data consistency.
• Lightweight alternative simulators: For pure obstacle avoidance algorithm verification, you can first iterate quickly on algorithms and reward design in lighter environments, then migrate to CARLA for final validation. Common alternatives include:
  • highway-env: An extremely lightweight 2D highway driving environment running at tens of thousands of FPS, suitable for rapid verification of decision-layer algorithms
  • MetaDrive: A 3D simulator designed specifically for autonomous driving RL, running much faster than CARLA (typically 10-50x faster) while maintaining reasonable driving scenario complexity
  • SUMO: A microscopic traffic simulator focused on traffic flow simulation, suitable for multi-agent traffic scenario research

MetaDrive is a particularly noteworthy choice. Developed by teams at UCLA and Shanghai AI Lab, built on the lightweight Panda3D engine, MetaDrive can simultaneously run dozens of parallel environment instances on a single machine without requiring GPU rendering. It supports procedurally generated road scenarios (including intersections, roundabouts, merges, splits, and various other topological structures), provides rich driving scenario diversity, and natively integrates the Gymnasium interface for seamless connection with RL libraries like SB3. MetaDrive is ideal for conducting extensive training experiments and hyperparameter searches under limited hardware conditions, and multiple papers at top conferences such as ICML, NeurIPS, and CoRL have used MetaDrive as their primary experimental platform.

A recommended workflow is: first rapidly iterate on reward function design and algorithm comparison in MetaDrive (a complete experimental round might take only a few hours), then once the best configuration is determined, migrate to CARLA for final high-fidelity validation and demonstration.

Summary: Core Recommendations for RL Obstacle Avoidance in CARLA

This undergraduate research project touches on three fundamental questions about reinforcement learning for autonomous driving obstacle avoidance, and the thinking direction is correct. Here are the consolidated recommendations:

• Algorithm level: DQN is reasonable as a discrete baseline, but it's strongly recommended to add SAC (continuous actions) as a comparison, using experiments to verify the value of continuous control. PPO can serve as a third comparison point to demonstrate on-policy vs. off-policy differences. Switching between all three in SB3 has very low cost.
• Reward design: This is the key factor determining project success or failure. Use dense rewards, carefully balance progress and safety, and watch out for the classic "passive stopping" trap. It's advisable to first debug the reward function in simple scenarios, confirm the agent exhibits correct behavioral tendencies, then increase scenario complexity.
• Simulation environment: CARLA is suitable for final validation and results presentation, but if training efficiency is limited, first rapidly iterate on algorithms and reward design in lightweight environments like MetaDrive, then migrate the optimal configuration to CARLA for validation.

The application of reinforcement learning in autonomous driving is far from mature—current industrial autonomous driving systems still primarily rely on rule engines and imitation learning, with RL remaining mostly in the research exploration stage. But it is precisely these solid comparative experiments and engineering practices that form the foundation for advancing the field. For undergraduate researchers, the value of completing the full pipeline of "environment setup → algorithm comparison → reward tuning → results analysis" far exceeds the final performance numbers themselves.