HELM Safety
A safety evaluation sub-suite within Stanford's HELM evaluation framework. It aggregates multiple datasets and behavioral dimensions to assess the safety of large models, covering safety attributes such as harmful refusal.
Timeline (last 90 days)
A psychometric audit study of the HELM Safety benchmark points out that similar total scores between models may mask entirely different attribute profiles.
The researchers selected four candidate datasets from HELM Safety that may target "harmful refusal", three of which are already saturated and cannot provide effective discriminating information.
HarmBench itself compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench's score together with those of other datasets into one top-line number.
All Facts (3)
A psychometric audit study of the HELM Safety benchmark points out that similar total scores between models may mask entirely different attribute profiles.
50%UnverifiedThe researchers selected four candidate datasets from HELM Safety that may target "harmful refusal", three of which are already saturated and cannot provide effective discriminating information.
50%UnverifiedHarmBench itself compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench's score together with those of other datasets into one top-line number.
50%