UtilityBench
A benchmark for evaluating the decision quality of language models when differences in error costs are taken into account. It contains instances annotated with utility values and supports evaluation of selective prediction.
Timeline (last 90 days)
The researchers built the UtilityBench benchmark, containing 15,000 instances with verifiable answers.
UtilityBench uses Best-Worst Scaling to obtain data-driven cost estimates, assigning a utility value to each instance.
In constructing UtilityBench, the researchers used BWS to have annotators compare the severity of different error consequences, thereby obtaining data-driven cost estimates that are comparable across instances.
All Facts (3)
UtilityBench uses Best-Worst Scaling to obtain data-driven cost estimates, assigning a utility value to each instance.
50%UnverifiedIn constructing UtilityBench, the researchers used BWS to have annotators compare the severity of different error consequences, thereby obtaining data-driven cost estimates that are comparable across instances.
50%UnverifiedThe researchers built the UtilityBench benchmark, containing 15,000 instances with verifiable answers.
50%