HarmBench
A benchmark designed to evaluate the safety of large language models, focusing on assessing a model's ability to resist harmful requests. It provides a standardized framework for testing model robustness against a range of potentially dangerous or policy-violating inputs.
Timeline (last 90 days)
HarmBench itself compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench's score together with those of other datasets into one top-line number.
Differential item functioning (DIF) analysis finds that models from different developers show systematic differences in HarmBench scores when their "refusal ability" is the same.
The results of multidimensional item response theory (MIRT) modeling show that HarmBench's scores strongly do not support the "single attribute" assumption; it simultaneously captures multiple distinct dimensions.
Several evaluation benchmarks released in 2024, such as JailbreakBench and HarmBench, have begun to incorporate assessments of multi-turn conversation scenarios.
Common alignment evaluation benchmarks include TruthfulQA and HarmBench.
Current mainstream evaluation benchmarks such as MMLU, TruthfulQA, and HarmBench are mostly designed for single dimensions and lack systematic measurement of value interaction effects.
obliteratus modified 841 out of 850 tensors; 44.8% of HarmBench responses never completed the thinking phase, making it the only variant to show significant degradation on intelligence tests.
orcarouter employs Arditi-style single-vector editing, modifying only 131 matrices at layer 38, achieving an attack success rate of 82.2% on the HarmBench test, with a copyright content unlock rate of 39%—the highest among all variants.
HarmBench is a standardized red-teaming evaluation framework created by a CMU team.
All Facts (8)
HarmBench is a red-teaming benchmark specifically targeting harmful content generation
70%UnverifiedHarmBench itself compresses different harmful behaviors into a single score, and HELM Safety's overall aggregation further folds HarmBench's score together with those of other datasets into one top-line number.
50%UnverifiedThe results of multidimensional item response theory (MIRT) modeling show that HarmBench's scores strongly do not support the "single attribute" assumption; it simultaneously captures multiple distinct dimensions.
50%UnverifiedDifferential item functioning (DIF) analysis finds that models from different developers show systematic differences in HarmBench scores when their "refusal ability" is the same.
50%UnverifiedSeveral evaluation benchmarks released in 2024, such as JailbreakBench and HarmBench, have begun to incorporate assessments of multi-turn conversation scenarios.
50%UnverifiedCommon alignment evaluation benchmarks include TruthfulQA and HarmBench.
50%UnverifiedCurrent mainstream evaluation benchmarks such as MMLU, TruthfulQA, and HarmBench are mostly designed for single dimensions and lack systematic measurement of value interaction effects.
50%UnverifiedHarmBench is a standardized red-teaming evaluation framework created by a CMU team.
50%