Are AI Safety Benchmarks Really Measuring "Harmful Refusal"? A Psychometric Audit

The "Aggregation Trap" in Safety Benchmarks

There is a widely overlooked methodological problem in AI safety evaluation: when a benchmark suite aggregates scores from multiple datasets and multiple behavioral dimensions into a single total score, what exactly is that total score measuring?

A psychometric audit of the HELM Safety benchmark points out that similar total scores across models may mask radically different attribute profiles. More troubling still, even focusing on the level of a single dataset does not guarantee that its score isolates one specific attribute. Using "harmful refusal"—the tendency of a model to reject dangerous or policy-violating prompts—as a candidate attribute, the researchers systematically tested whether it constitutes a single, independently measurable attribute within HELM Safety.

This work adopts a construct validity framework, whose core premise is: an attribute must first genuinely exist before a test can possibly measure it. This logic seems basic, yet it is frequently skipped in the practice of safety evaluation.

Construct validity is a core concept in psychometrics, first systematically proposed by Cronbach and Meehl in 1955. It asks a fundamental question: does a test really measure the psychological attribute or behavioral construct it claims to measure? The validation of construct validity is not a one-time event but an ongoing process of accumulating evidence, typically including content validity (whether items cover all aspects of the target construct), structural validity (whether scores exhibit an internal structure consistent with theoretical expectations), and external validity (whether scores have reasonable associations with other related measures). Applying this framework to AI safety evaluation means: before comparing models' safety scores, one must first demonstrate that these scores genuinely isolate and measure a clearly defined attribute; otherwise, score differences may stem from noise or confounding variables unrelated to the target attribute.

Four Candidate Datasets, Three Already Saturated

The researchers selected four candidate datasets from HELM Safety that might target "harmful refusal," and found that three of them were already saturated.

Saturation means that the space for score variation in these datasets is extremely limited—model performance tends to converge on these datasets, providing no effective discriminative information. This itself is one of the hidden dangers of aggregated scores: when a dataset's scores no longer discriminate, its weight in the total score remains, diluting the genuine signal carried by other datasets.

Only the HarmBench dataset made it into in-depth analysis. The researchers subjected it to two psychometric tests to determine whether the single attribute of "harmful refusal" truly underpins its scores.

Saturation in measurement theory refers to a test score losing its ability to discriminate among the objects being measured. When all models' scores on a dataset approach the ceiling (ceiling effect) or the floor (floor effect), the dataset's variance is extremely small and cannot provide effective information about differences in model capability. In the context of safety benchmarks, saturation typically occurs some time after a benchmark's release—as model developers optimize against known benchmarks, items that once had discriminative power are gradually "solved," and scores collectively climb toward near-perfect. If a saturated dataset is still included in an aggregate total score, its effect is equivalent to mixing a constant into the average, contributing no information while diluting the signal weight of other datasets that still discriminate.

Multidimensional Item Response Theory: HarmBench Is Not a Single Attribute

The first test used multidimensional item response theory (MIRT) modeling. The results showed that HarmBench scores strongly fail to support the "single attribute" assumption.

The core value of MIRT lies in its ability to detect whether a test is substantively measuring multiple latent dimensions. If a dataset truly measures only "harmful refusal tendency," then models' performance on its items should be explainable by a single latent variable. But HarmBench's data patterns indicate that it simultaneously captures multiple distinct dimensions—different harm types, different policy domains—each following different difficulty and discrimination parameters.

This means that when we say "Model A scores higher than Model B on HarmBench," this judgment is not equivalent to "Model A is more inclined than Model B to refuse harmful requests." Score differences may stem from entirely different subsets of items and behavioral patterns.

Item response theory (IRT) is a statistical framework for modeling the probability of a respondent's answers on test items. Its core idea is: each item has its own difficulty and discrimination parameters, and a respondent's performance on each item is determined by their latent ability level. Multidimensional item response theory (MIRT) is an extension of IRT that allows the model to simultaneously estimate multiple latent dimensions and test the extent to which each item is influenced by which dimension. Applying MIRT to AI safety evaluation can answer a key question: can models' scoring patterns across items be fully explained by a single latent variable (such as "harmful refusal tendency"), or are multiple dimensions needed to fit the data? If MIRT analysis shows that a unidimensional model fits significantly worse than a multidimensional model, it indicates that the dataset actually mixes multiple distinct behavioral attributes and cannot be regarded as a measurement tool for a single construct.

Differential Item Functioning Analysis: Same Ability, Different Scoring

The second test was differential item functioning (DIF) analysis. The researchers identified items where models from different developers, despite having the same "refusal ability," showed systematic differences in scores.

This directly undermines the fairness foundation of cross-model comparison. If two models have identical true refusal tendencies but benchmark scores differ due to item-level bias, then safety judgments based on these scores could mislead decision-making. The study also notes that when scope-specific matching was applied, most of these bias flags disappeared. This pattern is consistent with an aggregation effect—that is, the differences may stem from items mixing harmful behaviors from different domains, rather than genuine capability differences between developers. But the researchers also cautiously state that this is not sufficient to completely rule out developer differences in specific domains.

Differential item functioning (DIF) is a standard method in psychometrics for detecting test fairness. Classical DIF analysis focuses on: after controlling for respondents' true ability levels, are there systematic differences in the probability of answering the same item correctly among respondents from different groups (such as different genders or races)? If so, the item is biased against a certain group. In AI safety evaluation, researchers analogize "models from different developers" to different groups, and "refusal ability" to true ability level. When two models with comparable refusal ability show significantly different pass rates on a given item, that item is flagged as exhibiting DIF. The researchers further employed scope-specific matching—that is, matching ability only within subsets of items from the same harm type or policy domain—to test whether the bias was caused by items mixing behaviors from different domains.

What Aggregated Scores Hide

Zooming out, the problem becomes clearer. HarmBench itself already compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench together with other datasets' scores into one headline number.

The cost of this layer-by-layer aggregation is twofold: it hides the fact of dataset saturation and conflates essentially different behavior types. A model with a high refusal rate on violent content but a lenient stance in privacy-leakage scenarios, and a model with uniform performance across all scenarios, may receive similar total scores—but their safety implications are entirely different.

From this, the researchers put forward a methodological proposition: any safety score that averages across datasets and items must first demonstrate that it merits a "single attribute" interpretation before it can be used for model comparison. This requirement is not harsh; it merely applies the basic logic of construct validity to AI safety evaluation.

Broader Implications

For AI safety researchers and model developers, this audit delivers a clear signal: benchmark scores are not transparent windows but products of multiple layers of design decisions. The choice of datasets, the composition of items, the method of aggregation—each step shapes the final number and also shapes the safety judgments we make based on it.

At a time when the industry is racing to publish safety evaluation results, asking "what exactly is this score measuring" is more worthy of effort than asking "which model scores higher."

One-line Summary

HELM Safety benchmark's layered aggregation masks dataset saturation and attribute conflation, and its single-attribute interpretation lacks construct validity support.

Paragraph Summary

Starting from a construct validity framework, this study audited whether the HELM Safety benchmark truly measures the single attribute of "harmful refusal." The researchers found that three of four candidate datasets were already saturated, losing their ability to discriminate among models, yet still carrying weight in the total score. After in-depth analysis of the only usable dataset, HarmBench, multidimensional item response theory showed that its scores are driven by multiple latent dimensions rather than a single harmful refusal tendency; differential item functioning analysis further found systematic score biases among models from different developers with equivalent refusal ability, with most biases disappearing after scope-specific matching, suggesting the problem stems from items mixing different harm domains. The researchers argue that any aggregated safety score must first demonstrate that it merits a "single attribute" interpretation before being used for model comparison; otherwise, layer-by-layer aggregation only masks genuine behavioral differences.