FAITH: A New Framework for Safe Reinforcement Learning in High-Dimensional Systems

Safe reinforcement learning has long faced a dilemma: task performance and safety constraints are written into the same policy objective, and gradient updates may conflict with each other. Safety filters separate the two at the action execution level, which is a cleaner approach, but classical designs rely on analytical safety functions and dynamics models, while minimal intervention filters only focus on instantaneous action deviation and lack awareness of long-term returns. More critically, when no safe action exists, hard projection simply fails.

FAITH is a model-free framework proposed to address these gaps. According to the paper, it approximates the optimal state-action safety value and uses a feedforward network to amortize the minimal intervention filtering process. The task policy optimizes task return on the filtered dynamics, thereby recovering a feasible constrained problem without introducing competing safety terms into policy updates.

FAITH's Core Mechanism

Decoupling Safety Value from Control Policy

The key design of FAITH is that it does not stuff safety into the task policy's loss function, but instead learns a state-action safety value approximator. This approximator answers the question: how safe is it, in the long run, to execute a certain action in the current state. The filtering network then performs minimal intervention correction on actions based on this.

This is fundamentally different from traditional safety filters. Classical methods require explicit dynamics models and analytical safety functions, which are almost impossible to construct by hand in complex high-dimensional systems. FAITH takes a model-free route, delegating the filtering process to a feedforward network for amortization, bypassing the modeling bottleneck.

State-action safety value is a core concept in safe reinforcement learning, usually denoted as (Q^s(s,a)), which answers: "Given state s and action a, what is the expected cumulative safety cost over the entire future trajectory?" It is structurally symmetric to the task value (Q^r(s,a)), but its optimization objective is to minimize violation cost rather than maximize return. The classical constrained Markov decision process (CMDP) framework requires maximizing return while keeping cumulative cost within a threshold constraint, and the safety value is exactly the key quantity needed for the dual form of the constraint. FAITH approximates (Q^s) with a feedforward network, essentially a learned "brake judgment器": before an action enters the filter, it is evaluated by this network for long-horizon risk. Compared with approaches relying on analytical safety functions, this approximation can scale with data and is more robust to unknown dynamics, but its accuracy is limited by offline data coverage and generalization ability.

Minimal Intervention but Not Myopic

Standard minimal intervention filters only minimize the instantaneous deviation of actions, easily ignoring long-term task return. FAITH incorporates long-horizon information into the filtering decision through the safety value approximator, while keeping the intervention magnitude as small as possible. In other words, it neither over-intervenes in the task policy nor only cares about the immediate step.

This trade-off allows the task policy to continue optimizing task return on the filtered dynamics. According to the paper, this recovers a feasible constrained problem, and the task policy update contains no competing safety term—safety and performance no longer fight in the same gradient.

Minimal intervention filter is a class of safety mechanism that makes fine adjustments before actions enter the actuator: when the action given by the task policy is already safe, no modification is made; only when it is about to cross the boundary is a correction applied that is just sufficient to pull the action back into the safe set. Its advantage is that the damage to task performance is theoretically minimal, so it is often used as a "safety layer" stacked on top of an existing policy. Classical implementations such as quadratic programming filters based on control barrier functions (CBF) rely on analytical barrier functions and system dynamics. However, versions that only minimize instantaneous deviation only check whether the current action crosses the boundary and do not evaluate the action's future impact, so they often fail in scenarios requiring long-horizon avoidance such as early deceleration or detouring. FAITH introduces long-horizon factors into this minimization objective through the safety value, essentially adding a "lookahead" dimension to the instantaneous filter.

Degradation Strategy When No Safe Action Exists

Hard projection is undefined when no safe action is available, which is a fatal weakness of many safety filters. FAITH's approach is: when the learned safety condition is not satisfied by any action, the filter instead selects the action with the minimum predicted peak harm. This amounts to a principled degradation, rather than crashing outright or taking random actions.

Hard projection refers to forcing the task action onto the safe action set, typically implemented by solving an optimization problem with hard constraints. Mathematically, it requires the safe set to be non-empty. Once the feasible region is empty—that is, any action in the current state will cause a violation—the problem has no solution, and the program either errors out or degenerates into inexplicable behavior. This is common in scenarios approaching physical limits (such as a humanoid about to fall or a robotic arm entering a singular configuration). Soft constraint methods trade constraint relaxation for feasibility, but at the cost of allowing some violations. FAITH's "peak harm minimization" lies between the two: it does not pretend that a safe action exists, but acknowledges the lack of a solution and selects the action with the lowest harm upper bound. This is a feasibility-aware degradation approach that can still produce executable and interpretable decisions in the worst case.

Experimental Performance: From Double Integrator to Real Humanoid

Return and Violations in Benchmark Environments

In the double integrator example and Safety Gym environments, FAITH achieves the highest return among methods without feasible start violations, while matching the lowest harm level under infeasible starts. Achieving both metrics simultaneously shows that the trade-off between safety and performance is significantly alleviated.

Safety Gym is a safe reinforcement learning benchmark suite proposed by OpenAI, containing agents of varying complexity such as Point, Car, and Ant. Tasks are scattered with hazards to avoid and goals to reach, while "feasible/infeasible start" distinguishes whether the policy is within the safety boundary from the beginning. It has become a commonly used benchmark because it exposes both return and violation cost curves, directly testing a method's trade-off between performance and safety. Feasible start means the initial state satisfies all safety constraints, while infeasible start means the agent is in violation from the outset, forcing the method to demonstrate recovery ability rather than merely avoiding violations. FAITH achieves high return and low harm on these two types of starts respectively, indicating that its filtering mechanism remains stable under boundary conditions.

Safety Rate of a 29-DOF Humanoid

On a 29-DOF humanoid system, FAITH achieves a 99.95% safety rate in the Walking-Avoid task while retaining 97% of the unfiltered return. This number is worth breaking down: the safety rate is extremely high, but the return loss is very small, indicating that the filter does not suppress the task policy into a conservative standstill.

In the Push-Avoid task, FAITH achieves the highest measured safety rate. The paper mentions that it learned to sacrifice balance and fall in the direction away from the protected area. This strategy may appear aggressive from a traditional safety perspective, but it buys a higher safety rate—the fall direction is the result of a trade-off.

Real Robot Validation

The same policy was also deployed for demonstration on a real-world Unitree G1 humanoid robot. Transfer from simulation to real hardware is a key threshold for the deployment of safe reinforcement learning methods, and this validation adds practical weight to the framework.

A Brief Analysis

FAITH's idea is to "let safety be safety and tasks be tasks," then bridge the two with a learned safety value. This is more reasonable than weighted summation in a single objective, and more scalable than purely analytical safety filters.

There are still questions worth asking. The accuracy of the safety value approximator directly determines filtering quality, and the distribution of approximation error in high-dimensional states is unclear. Peak harm minimization when no safe action exists is a heuristic remedy, and how strong its theoretical guarantees are is not elaborated in the paper abstract. The specific task scope and safety boundaries of the real robot demonstration also require reading the main text for details.

For researchers working on embodied intelligence and safe control, FAITH provides a new baseline worth reproducing and comparing against: model-free, feasibility-aware, supportive of high-dimensional systems, and not dependent on analytical safety functions.

One-line Summary

FAITH uses a model-free safety value approximator to bridge safety filtering and task policy, achieving both safety and performance in high-dimensional systems.

Paragraph Summary

FAITH is a model-free framework proposed for the problem of "conflicting gradients between safety and performance" in safe reinforcement learning. Its core idea is to decouple safety filtering from the task policy: use a feedforward network to approximate the state-action safety value, perform minimal intervention correction on actions based on it, and let the task policy continue optimizing return on the filtered dynamics, thereby avoiding introducing competing safety terms in the same loss function. Compared with classical filters that rely on analytical safety functions and dynamics models, FAITH requires no modeling and can scale to high-dimensional systems; compared with minimal intervention filters that only look at instantaneous deviation, it incorporates long-horizon information into decision-making. When no safe action is available, it selects the action with the minimum predicted peak harm as a degradation strategy, rather than letting hard projection fail. Experiments show that it achieves high return and low violations in the double integrator and Safety Gym, reaches a 99.95% safety rate on a 29-DOF humanoid while retaining 97% of the unfiltered return, and has been validated on a real Unitree G1 robot.