Reward Shaping and RL Debugging in Practice: A Systematic Guide from Oscillation to Convergence

Starting with a Drone Project

Recently, a student studying in France shared their struggles with a reinforcement learning (RL) project on Reddit, striking a chord with many practitioners.

Reinforcement Learning Background: Reinforcement learning is one of the three major paradigms in machine learning, rooted in the operant conditioning theory from behavioral psychology. Unlike supervised learning (which requires labeled data) and unsupervised learning (which discovers intrinsic data structure), RL learns optimal policies through continuous interaction between an agent and its environment. Its mathematical foundation is the Markov Decision Process (MDP), defined by state space S, action space A, transition function P, reward function R, and discount factor γ. In recent years, the combination of deep neural networks with RL (Deep RL) has achieved breakthroughs in complex continuous control scenarios such as gaming, robotic control, and autonomous driving. DeepMind's AlphaGo and OpenAI's competitive game agents are landmark achievements in this space.

The student's goal was to guide an FPV drone via vision-based navigation to reach a series of waypoints or track a moving target. FPV (First Person View) drones typically carry cameras that stream live footage, widely used in racing, aerial photography, and military reconnaissance. PX4 is the most mainstream open-source flight control software stack, running on hardware platforms like Pixhawk, with built-in attitude estimation, position control, motor mixing, and other multi-layer control loops. The full control chain is: guidance policy → output acceleration commands → PX4 flight controller execution. In this hierarchical architecture, the RL policy acts as a "guidance layer," outputting desired acceleration or velocity commands, while PX4's inner-loop PID controller handles execution. This hierarchical control design decouples high-level decision-making from low-level execution, but also introduces potential risks of inter-layer latency and command incompatibility — a common cause of Sim-to-Real transfer failures.

The student initially built a Proportional Navigation Guidance Law (PNG) using traditional methods, which performed well in simulation but degraded noticeably after switching to camera-based observation inputs. PNG is a classic algorithm derived from missile guidance. Its core idea is that control commands are proportional to the line-of-sight angular rate of the target, expressed as a = N * V * ω_LOS, where N is the navigation ratio (typically 3–5), V is flight speed, and ω_LOS is the line-of-sight angular rate. PNG is mathematically provably optimal for tracking a target moving at constant velocity in a straight line, with minimal computation and excellent real-time performance. However, when sensor noise (such as camera measurement error) is introduced, the estimation of ω_LOS picks up high-frequency noise, causing control command jitter — the direct cause of performance degradation after switching to camera input.

He then turned to an RL approach, replacing the original guidance law with an RL policy. The pipeline "technically worked" in simulation, but a host of bizarre behaviors emerged: persistent oscillations, abuse of Bang-Bang control (where commands repeatedly jump between extreme values), and other issues kept piling up.

He repeatedly modified the reward function, but each revision triggered new unexpected problems rather than fixing the original one. This is the classic dilemma nearly every RL practitioner encounters — the black-box problem of reward shaping and policy debugging. This article uses this real-world case to outline a systematic troubleshooting approach.

Why Is Reinforcement Learning Debugging So Difficult

Unlike supervised learning, reinforcement learning has no direct "ground truth" — it relies on reward signals to indirectly guide policy generation. This creates two fundamental challenges.

The Causal Chain Between Reward and Behavior Is Long

An agent's final behavior is the result of coupled interactions among the reward function, environment dynamics, exploration strategy, network architecture, and more. When you observe oscillations, the root cause could be a poorly designed reward, the action space definition, observation noise, or even response latency in the PX4 low-level controller. This "one effect, many causes" characteristic means that simply staring at the reward function and tweaking parameters over and over is often inefficient.

Reward Hacking Is Everywhere

The "Bang-Bang command abuse" in this case is a classic example of reward hacking. Bang-Bang control has legitimate uses in control theory — Pontryagin's Maximum Principle proves that the optimal solution to certain time-optimal control problems is precisely Bang-Bang in form. However, in RL, Bang-Bang behavior typically arises from pathological reward signals: when the agent discovers that higher immediate reward increments can be obtained at the boundaries of the action space, and the reward function imposes no penalty on control cost, gradients continuously push the policy toward action boundaries. High-frequency switching not only consumes motor lifespan rapidly but can also excite the airframe's resonance frequency, potentially causing structural damage on real hardware. This class of phenomena is broadly referred to as "Reward Hacking" in the research community and is the most concrete engineering manifestation of the RL alignment problem.

When the reward function only cares about "whether the agent is getting closer to the target" without any control smoothness constraints, the agent discovers that high-frequency, large-amplitude oscillations can approach the target faster — exploiting a loophole in the reward function. This isn't a bug; it's the policy "over-optimizing" the reward you actually wrote, rather than the behavioral objective you truly intended.

Practical Principles for Reward Shaping

For continuous-control guidance tasks, reward shaping should follow several core principles.

Build in Layers, from Sparse to Dense

Start with the simplest sparse reward (e.g., "reach waypoint +1") to verify that the entire pipeline functions correctly, then gradually add dense shaping terms. Dense rewards typically include:

• Distance term: Negative value of distance to target or its rate of change, guiding the agent to continuously close in;
• Smoothness term: Penalize the action change rate |a_t - a_{t-1}| to directly suppress oscillations and Bang-Bang behavior;
• Attitude/energy term: Penalize excessively large accelerations or unreasonable attitudes, aligning with real flight constraints.

Use Potential-Based Shaping

A technique theoretically proven to "not alter the optimal policy" is potential-based reward shaping (Ng et al., 1999). Its form is F = γΦ(s') - Φ(s), where Φ is typically the negative distance to the goal.

The core guarantee of this theorem is "policy invariance": if the shaping reward F satisfies the above form, then the set of optimal policies on the original MDP is identical to that of the new MDP after adding F. This means engineers can freely use potential functions to accelerate training convergence without worrying about introducing spurious optimal policies. For guidance tasks, the negative distance Φ(s) = -d(s, goal) has an intuitive physical meaning — "states closer to the goal are more valuable" — effectively guiding agents to continuously move toward the target in sparse-reward environments. It is the preferred approach for guidance-type tasks.

Change Only One Variable at a Time

The root cause of "every revision triggers new problems" in this case is often that multiple reward term weights are adjusted simultaneously. Develop the habit of baseline configuration + single-variable controlled experiments: change only one coefficient at a time, record the corresponding behavioral changes, and gradually build up an intuitive mapping of "parameter → behavior."

Systematic Debugging Strategies

Beyond reward design itself, a structured debugging process can dramatically improve the efficiency of root cause identification.

Isolate and Validate Each Module

First, confirm whether the downstream pipeline "policy → acceleration → PX4" is correct on its own. Use a known-effective deterministic policy (such as the proportional navigation law mentioned earlier) to replace the RL policy and verify that downstream execution is faithful. If even a correct policy behaves abnormally, the problem lies at the environment or interface level, not in RL itself.

Monitor Key Training Metrics

Beyond the reward curve, the following metrics are also worth close attention:

• Action distribution: If actions are heavily concentrated at boundary values, Bang-Bang hacking has almost certainly occurred;
• Value function estimates: Whether Value Loss converges reflects whether the critic has learned a meaningful signal;
• Policy entropy: In Deep RL, policy entropy measures the uncertainty of the action distribution and is a direct quantification of exploration capacity. Premature entropy collapse is a common pathological phenomenon in training — when the policy becomes highly deterministic before sufficiently exploring the state space, causing the agent to fall into a local optimum. SAC (Soft Actor-Critic) systematically addresses this by explicitly incorporating an entropy regularization term into the optimization objective, while PPO (Proximal Policy Optimization) softly encourages exploration via the entropy coefficient (entropy_coef). Premature entropy collapse implies insufficient exploration and the policy may settle into suboptimal behavior;
• Episode length and success rate: More intuitively measure task completion quality than cumulative reward.

Use Visual Replay to Pinpoint Issues

Drawing curves alone is often insufficiently intuitive. Replaying the agent's trajectories and action sequences — overlaying drone position, target position, and action commands on the same time axis — often makes the frequency of oscillations and the triggering moments of Bang-Bang behavior immediately apparent.

Additional Considerations for Sim-to-Real Transfer

The ultimate goal of this case is deployment on a real FPV drone, which raises the well-known Sim-to-Real Gap problem in RL.

The root cause of the Sim-to-Real Gap lies in the imperfect approximation of the real physical world by simulation models (Model Mismatch). Specific sources include: aerodynamic simplifications (simulators typically ignore complex phenomena like turbulence and ground effect), distorted sensor noise models, unmodeled motor response delays, and domain gaps between rendered visuals and real images. A policy that "technically works" in simulation may well fail on real hardware due to the cumulative effect of these factors.

A common countermeasure is Domain Randomization — randomly perturbing parameters such as mass, latency, and observation noise during training to force the policy to learn more robust behavior. This approach was popularized by teams at OpenAI and ETH Zurich, and has demonstrated significant effectiveness on tasks such as dexterous hand manipulation and quadruped locomotion. In addition, System Identification and Adaptive Control are common complementary approaches that refine simulation model parameters by collecting small amounts of data on real hardware.

The action smoothness penalty mentioned earlier is especially critical in real-hardware scenarios, because real motors simply cannot handle the high-frequency command shocks of Bang-Bang style control.

Conclusion: You're Optimizing the Reward You Wrote, Not the Goal You Want

This student's predicament is a rite of passage for almost every RL practitioner. The core insight can be distilled into one sentence: An RL agent always optimizes the reward you wrote, not the goal you truly intend. Rather than treating the reward function as a mystical recipe and repeatedly guessing at it, build the engineering debugging habit of "module isolation validation + single-variable experiments + behavior visualization." When oscillations and hacking behaviors appear, they are not enemies — they are the most honest feedback signals about flaws in your reward design.

Key Takeaways