RESCUE
A counterfactual evaluation framework for runtime fault monitoring in robotics. It uses simulation replay to compare the causal effects of different intervention actions, shifting the evaluation focus from detection accuracy to the practical value of actions triggered by alerts.
Timeline (last 90 days)
RESCUE is a counterfactual evaluation framework proposed to address blind spots in robotic failure monitoring assessment, shifting the core focus from 'whether detection is accurate' to 'whether responding to an alarm can truly lead to useful intervention'.
RESCUE restores the simulator state and policy state at various checkpoints during execution, identifies the latest checkpoint that satisfies specific recovery criteria, and then evaluates whether taking action based on the alarm truly reduces the expected cost of task completion.
The RESCUE study generated a total of 128,340 counterfactual branches.
The recoverability of a robot task is not determined solely by the robot's current state, but depends on the pairing of the state with the type of intervention.
Introducing recovery actions reduced recoverability under certain conditions, meaning scripted recovery is not always a net positive.
Under the primary strong-policy condition, a good detection score does not demonstrate a lower intervention cost.
Under the 'stop and reset' model, even an idealized perfect detector can only tolerate a false positive rate of 2.0% to 3.3%; once this range is exceeded, the cost of unnecessary resets outweighs the savings from early termination of failing tasks.
When building monitors, the weight assigned to precision should be significantly increased, rather than optimizing purely for recall.
The value of a monitor depends on the available intervention actions and the workload into which it is deployed, and cannot be measured independently of the intervention actions and the deployment workload.
All Facts (9)
RESCUE is a counterfactual evaluation framework proposed to address blind spots in robotic failure monitoring assessment, shifting the core focus from 'whether detection is accurate' to 'whether responding to an alarm can truly lead to useful intervention'.
50%UnverifiedRESCUE restores the simulator state and policy state at various checkpoints during execution, identifies the latest checkpoint that satisfies specific recovery criteria, and then evaluates whether taking action based on the alarm truly reduces the expected cost of task completion.
50%UnverifiedThe RESCUE study generated a total of 128,340 counterfactual branches.
50%UnverifiedThe recoverability of a robot task is not determined solely by the robot's current state, but depends on the pairing of the state with the type of intervention.
50%UnverifiedIntroducing recovery actions reduced recoverability under certain conditions, meaning scripted recovery is not always a net positive.
50%UnverifiedUnder the primary strong-policy condition, a good detection score does not demonstrate a lower intervention cost.
50%UnverifiedUnder the 'stop and reset' model, even an idealized perfect detector can only tolerate a false positive rate of 2.0% to 3.3%; once this range is exceeded, the cost of unnecessary resets outweighs the savings from early termination of failing tasks.
50%UnverifiedWhen building monitors, the weight assigned to precision should be significantly increased, rather than optimizing purely for recall.
50%UnverifiedThe value of a monitor depends on the available intervention actions and the workload into which it is deployed, and cannot be measured independently of the intervention actions and the deployment workload.
50%