UtilityBench: How Should We Evaluate Language Models When Error Costs Differ?

Mainstream language model evaluations typically treat all errors equally, caring only about whether the answer is correct while ignoring the enormous variation in the cost of errors across different scenarios. In high-stakes domains such as healthcare, a single mistake can be life-threatening. To address this, researchers have proposed a utility-based evaluation paradigm and built the UtilityBench benchmark and the UtilityAUC metric, designed to measure a model's decision quality when the consequences of errors are taken into account.

Why a New Evaluation Paradigm Is Needed

Traditional evaluation protocols, such as accuracy, treat all errors as equivalent. But in real-world applications, the consequences of errors vary enormously: in medical diagnosis, a misdiagnosis can lead to serious outcomes, whereas in general question answering, the impact of an error is far smaller. This one-size-fits-all evaluation approach cannot reflect a model's utility in real-world scenarios.

The researchers point out that the impact of language model errors goes far beyond what correctness can capture. To this end, they propose a utility-based evaluation paradigm that explicitly incorporates the different consequences of errors. This paradigm guides models to choose between answering and abstaining when uncertain, thereby making decisions that match the importance of the scenario.

Selective prediction refers to a model's ability to abstain when uncertain, handing the sample over to humans or downstream processes. This mechanism is especially important in high-stakes scenarios such as healthcare and finance, because the cost of abstaining is usually lower than the cost of giving a wrong answer. The utility-based evaluation paradigm combines selective prediction with error consequences: the model must decide not only "what to answer" but also "whether to answer at all." When the cost of error is high, abstaining may be the better strategy; when the cost of error is low, answering is more worthwhile. This decision framework shifts evaluation from mere correctness toward expected utility maximization.

UtilityBench: A Large-Scale Benchmark for Quantifying Error Severity

To quantify error severity, the research team built UtilityBench, which contains 15,000 instances with verifiable answers. They used Best-Worst Scaling to obtain data-driven cost estimates, thereby assigning a meaningful utility value to each instance.

These quantified utility values shape the policy for when a model should answer or abstain, guiding the model to make decisions consistent with the true importance of each scenario. For example, in scenarios where the cost of error is high, the model may be more inclined to abstain to avoid potentially costly mistakes.

Best-Worst Scaling (BWS) is a preference elicitation method commonly used in psychometrics. In traditional subjective rating, annotators often struggle to give stable absolute scores, whereas BWS asks annotators to select the "best" and "worst" options from a set, deriving relative utility values for each option through repeated comparisons. This method effectively reduces rating bias, improves discriminability, and places lower demands on annotator consistency. In constructing UtilityBench, the researchers used BWS to have annotators compare the severity of different error consequences, thereby obtaining data-driven cost estimates that are comparable across instances and assigning a meaningful utility value to each instance.

UtilityAUC: Utility Preference Alignment Beyond Error Frequency

In addition to the benchmark, the researchers introduced the UtilityAUC metric, which measures the area under the utility curve determined by the model's response ranking. Unlike traditional accuracy or AURC (Area Under the Risk-Coverage Curve), UtilityAUC captures how well a model aligns with utility preferences, not merely error frequency.

Theoretical analysis shows that UtilityAUC can distinguish between models that perform similarly on standard accuracy and AURC. This means that even if two models have the same accuracy, their performance on high-cost errors may differ markedly, and UtilityAUC can reveal this difference.

AURC (Area Under the Risk-Coverage Curve) is a classic metric in the field of selective prediction. It characterizes how a model's risk (error rate) changes as coverage decreases when the model is allowed to rank by confidence and progressively abstain. AURC focuses on error frequency but treats all errors as equivalent. UtilityAUC builds on this by introducing utility weights, giving high-cost errors greater weight in the curve integral. Therefore, two models with the same AURC can have significantly different UtilityAUC values if their error distributions on high-cost samples differ—this is precisely why UtilityAUC can distinguish models that have "the same accuracy but different utility."

Experimental Findings: High Accuracy Does Not Equal High Utility

The research team evaluated 20 state-of-the-art models and found that higher accuracy or more advanced architectures do not always lead to higher UtilityAUC. This finding highlights the gap between problem-solving capability and responsible decision-making.

In other words, a model that scores highly on traditional benchmarks may perform poorly when weighing the cost of errors. This suggests that when deploying language models, we need to pay closer attention to their performance at the utility level, not just their accuracy.

Implications for Industry

The introduction of UtilityBench and UtilityAUC offers a new perspective on language model evaluation. In high-stakes domains such as healthcare, finance, and law, this utility-based evaluation approach can help developers choose more responsible models and guide models to make safer decisions when uncertain.

As language models are increasingly deployed in the real world, this evaluation paradigm that accounts for the cost of errors is expected to become standard practice, driving models from "correct" toward "responsible."

One-line Summary

Researchers propose UtilityBench and UtilityAUC to evaluate language model error costs through utility.

Paragraph Summary

Traditional language model evaluation treats all errors equally and focuses only on accuracy, ignoring the enormous variation in error costs across scenarios. To address this, researchers propose a utility-based evaluation paradigm, build the UtilityBench benchmark with 15,000 instances, quantify error severity through Best-Worst Scaling, and introduce the UtilityAUC metric to measure the area under the utility curve determined by model response ranking. After evaluating 20 advanced models, they find that higher accuracy or more advanced architectures do not always yield higher UtilityAUC, highlighting the gap between problem-solving capability and responsible decision-making. This paradigm is expected to drive models in high-stakes domains from "correct" toward "responsible."