Open-Source Robustness Testing Framework SIRFEX: Why Does Failure-Driven PPO Retraining Hurt Nominal Performance?

A developer is building an open-source framework called SIRFEX (Simulation-based Integrity, Robustness & Failure Exploration), designed to stress-test autonomous controllers in simulation, discover reproducible failures, and evaluate whether proposed improvements actually enhance robustness. However, preliminary experiments he shared on Reddit revealed a counterintuitive phenomenon: fine-tuning a PPO policy on discovered failure scenarios actually made the controller perform worse under normal operating conditions. The developer chose to disclose these negative results and seek community input, offering a case worth examining in depth for the RL control field.

The SIRFEX Framework: From Failure Discovery to Robustness Verification

SIRFEX is positioned as a complete simulation testing loop. According to the original post, it already implements deterministic simulation, scenario search, failure discovery and replay, and a physical feasibility analyzer—used to distinguish "failures the controller could have avoided" from "physically impossible operating conditions." On the training side, the framework supports PPO training, compute-matched continuation experiments, and failure-driven curricula with validation stress exposure. There is also an optional LangGraph/Groq research agent for proposing bounded training intervention suggestions.

The newly added robustness testing layer includes signed constraint margins, differential controller testing, and perturbation-to-failure analysis. The experimental environment is a simplified pump/tank control system, with comparisons against Stable-Baselines3 PPO, an analytical controller, and a rule-based baseline. The entire project runs locally on CPU, emphasizing reproducibility and scientific honesty.

PPO (Proximal Policy Optimization) is a commonly used policy gradient algorithm in reinforcement learning that ensures training stability by limiting the magnitude of each policy update. It uses a clipped surrogate objective function to prevent the new policy from deviating too far from the old one, thereby balancing sample efficiency and training stability. PPO is widely used in continuous control tasks such as robot control and autonomous driving. In SIRFEX, PPO is used to train the pump/tank controller's policy, and failure-driven fine-tuning involves continuing training on specific scenario data from an existing PPO policy. Understanding PPO's clipping mechanism helps analyze why stress samples distort the optimization objective: when the proportion of stress samples increases, the gradient contribution of normal samples is diluted, and the policy may overfit to the stress distribution, leading to degraded nominal performance.

Negative Results: Stress Exposure Improves Robustness but Sacrifices Nominal Performance

The developer's core finding comes from a 51,200-interaction development study. He originally expected that training PPO on discovered failure scenarios would improve robustness. But that's not what happened: increasing stress exposure did improve tracking performance under difficult conditions, but frequently caused tracking degradation under normal conditions.

Taking seed 3 as an example:

Training strategy	Nominal MAE	Stress MAE
Normal data only	0.0177	0.1166
5% stress	0.0296	0.0654
10% stress	0.0501	0.0555

The data clearly shows a trade-off: stress training reduced the controller's MAE under stress scenarios from 0.1166 to 0.0555, but nominal-scenario MAE worsened from 0.0177 to 0.0501. More critically, the controller trained on normal data only had already achieved zero physical violations on this development stress set, meaning "improved stress tracking" does not equate to "improved safety."

The developer also tried using GPT-OSS 120B to propose a retraining curriculum. In a matched development pilot on one seed, this approach performed worse than a simple fixed 5% stress strategy, and neither passed the nominal performance preservation guardrail. Further diagnosis showed that upper action saturation intensified as stress exposure increased, suggesting the policy may drift toward more aggressive pumping behavior. But the developer has not yet determined whether this is due to catastrophic forgetting, reward incentives, curriculum ordering, or other causes.

MAE (Mean Absolute Error) is a common metric for measuring tracking performance, calculated as the average of absolute errors between predicted and true values. In control systems, a smaller MAE indicates the controller output is closer to the setpoint. In this article, nominal MAE refers to the mean absolute error under normal operating conditions, and stress MAE refers to the mean absolute error under stress scenarios (such as disturbances or extreme conditions). MAE's linear penalty for errors makes it less sensitive to outliers than mean squared error, making it suitable for evaluating control system smoothness. The table data shows that as the proportion of stress training increases, nominal MAE rises from 0.0177 to 0.0501, meaning tracking accuracy under normal conditions significantly degrades; while stress MAE drops from 0.1166 to 0.0555, indicating improved performance in stress scenarios. This trade-off is a typical nominal-robust performance compromise in robustness research.

Several Directions the Community Might Focus On

The developer raised three specific questions, which also point to common difficulties in RL robustness research.

How can robustness be improved without sacrificing nominal performance? He listed candidate approaches such as nominal rehearsal, policy anchoring, and constrained RL. From the negative results, simply increasing the proportion of stress samples distorts the policy's optimization objective—the gradient signal from normal conditions is diluted, and the policy may overfit to the stress distribution. Nominal rehearsal (mixing a fixed proportion of normal data into each training round) and policy anchoring (using KL divergence to constrain the new policy from deviating from the old one) are intuitive mitigation measures, but they require experimental validation.

When the constraint violation frequency has already saturated at zero, how do you measure meaningful robustness? This is a metric design problem. Zero violations mean the current test set can no longer distinguish controller quality. The developer is moving toward quantitative robustness margins and differential testing, which is a reasonable direction—signed constraint margins can reflect how far the controller is from the violation boundary, and differential testing can compare behavioral differences between controllers under the same perturbations.

Are there existing benchmarks or testing methods for comparison? In the control field, safety reinforcement learning benchmarks (such as Safety Gym) and robustness testing methods (such as adversarial perturbation-based testing) may provide references. However, SIRFEX's physical feasibility analysis layer is unique, and how to align it with general benchmarks still needs exploration.

Constrained reinforcement learning (Constrained RL) is a class of reinforcement learning methods that satisfy safety constraints while optimizing cumulative reward. Unlike standard RL, constrained RL incorporates constraint violations (such as physical violations or safety thresholds) as part of the optimization objective, typically using Lagrangian methods or constrained policy optimization to balance reward and constraints. In the context of SIRFEX, constrained RL can be used to ensure the controller improves robustness without sacrificing nominal performance—for example, treating nominal MAE degradation as a constraint to limit the direction of policy updates. The "policy anchoring" listed by the developer is essentially a simplified form of constrained RL, constraining how far the new policy deviates from the old one via KL divergence. The challenges of constrained RL lie in the trade-off between constraint satisfaction and reward maximization, as well as the reasonable setting of constraint thresholds.

Why Failure-Driven Fine-Tuning May Be Worse Than Ordinary Continuation Training

In the original post, the developer mentioned that failure-driven PPO fine-tuning performed worse than ordinary continuation training in the pilot. This phenomenon is not uncommon in RL practice, and possible reasons include:

• Distribution shift: Failure scenarios are typically concentrated in edge regions of the state space, and fine-tuning on these samples causes the policy to over-optimize edge performance while neglecting fine-grained control in central regions.
• Side effects of reward reshaping: If the failure-driven curriculum changes the reward function or sample weights, the policy may learn "gaming" behavior, such as more aggressive pumping to quickly correct deviations, at the cost of smoothness under normal conditions.
• Curriculum order sensitivity: The timing and proportion of stress exposure introduction may affect the final policy. The fixed 5% and 10% stress proportions used by the developer may not be the optimal schedule.

The developer observed intensified upper action saturation, consistent with the hypothesis of "policy drift toward aggressive pumping." In a pump/tank system, more aggressive pumping may help quickly track pressure setpoints but introduces greater overshoot and oscillation, thereby worsening nominal MAE.

Catastrophic forgetting is a common phenomenon in neural networks when learning multiple tasks sequentially: learning a new task causes significant performance degradation on old tasks. In reinforcement learning, when a policy is fine-tuned on failure scenarios, network parameters may undergo substantial adjustments to adapt to the new distribution, thereby forgetting the fine-grained control strategies learned under normal conditions. The developer observed nominal performance degradation, and catastrophic forgetting is one candidate explanation. Mitigation methods include experience replay (mixing old data), elastic weight consolidation (limiting updates to important parameters), and policy anchoring (constraining policy deviation). However, catastrophic forgetting may interact with distribution shift, reward reshaping, and other factors, requiring ablation experiments to distinguish.

Implications for Open-Source RL Testing Frameworks

SIRFEX's negative results are valuable in themselves. They remind us that failure-driven training is not a panacea for robustness. When constraint violations are already zero, continuing to fine-tune on failure scenarios may simply be optimizing an already-saturated metric at the cost of normal performance. The developer's shift toward quantitative margins and differential testing is a direction more likely to yield discriminative power.

For teams building similar testing frameworks, this case offers several takeaways:

1. Nominal performance guardrails should be treated as hard constraints: Any robustness improvement scheme should be evaluated under the premise that nominal performance does not significantly degrade.
2. Failure scenario sampling needs to be balanced with normal scenarios: Simply increasing the stress proportion may backfire, requiring smarter curriculum scheduling or constraint mechanisms.
3. Robustness metrics should go beyond violation frequency: When violation frequency saturates, metrics such as margins, recovery time, and perturbation sensitivity better reflect true robustness.
4. Negative results are worth publishing: By disclosing negative results and experimental details, the developer provides the community with a reproducible reference, which is more helpful to field progress than reporting only success cases.

SIRFEX is still in its early stages, with code open-sourced on GitHub. The developer explicitly welcomes criticism, related papers, implementation suggestions, and contributions. For researchers focused on RL control, safe reinforcement learning, and simulation testing, this is a project worth following.

One-line Summary

Open-source framework SIRFEX finds that failure-driven PPO retraining improves stress performance but degrades nominal performance, revealing robustness training trade-offs.

Paragraph Summary

SIRFEX is an open-source simulation testing framework for stress-testing autonomous controllers, discovering reproducible failures, and evaluating improvement schemes. The developer experimented with PPO on a pump/tank control task and found that increasing the proportion of stress scenario training improved tracking performance under difficult conditions but significantly degraded nominal performance under normal conditions, creating a clear trade-off. A controller trained on normal data only had already achieved zero physical violations on the development stress set, showing that improved stress tracking does not equal improved safety. A retraining curriculum proposed by GPT-OSS 120B performed worse than a fixed 5% stress strategy in the pilot, and neither passed the nominal performance guardrail. The developer is shifting toward quantitative robustness margins and differential testing to continue measuring robustness after violation frequency saturates. These negative results provide a reproducible reference for RL robustness research.