EpiSEM-Bench

EpiSEM-Bench is an executable benchmark covering six domains and containing 300 matched unsafe/safe task pairs.

In each task pair of EpiSEM-Bench, the tasks share the same user request, tool interface, and most of the environment state, with the only controlled difference being whether a certain key safety fact is included in the tool response.

EpiSEM-Bench is equipped with an independent auditing tool for testing whether agents proactively seek missing facts and whether, once the facts are supplied, they can complete the legitimate task while refusing the forbidden action.

The benchmark reveals significant differences among ten models across the three dimensions of robustness to omission, evidence acquisition, and appropriate action.

Resistance to missing evidence does not necessarily translate into useful recovery capability.

Restoring access (i.e., allowing agents to obtain the missing facts) reduced the unsafe execution rate of all ten models.

This type of failure (unsafe execution caused by omission of a key fact) cannot be eliminated through adversarial training or fact-checking, because it involves no false signal, only an information gap.

Evaluation in EpiSEM-Bench not only gives binary safe/unsafe verdicts but further determines the minimal failure base observed, including failures that still occur when the evidence is complete and cases where no failure is observed even after exhaustive testing.

In EpiSEM-Bench, the paired design allows the benchmark to compute a "conditional unsafe rate", i.e., the incremental failure of a model on the missing version after controlling for within-pair similarity, which better reflects the model's sensitivity to evidence gaps than the global unsafe rate.

A single unsafe execution rate metric masks capability differences among models, so evidence acquisition behavior and post-recovery decision quality need to be reported as well.