Escalation Benchmark
An AI model evaluation benchmark designed by developer Sean Campbell. It specifically measures whether a model can recognize its own limitations when faced with unanswerable questions and choose to escalate (ESCALATE) rather than force an answer, focusing on metacognition and calibration.
Timeline (last 90 days)
Developer Sean Campbell designed an evaluation framework called the Escalation Benchmark, which measures whether a model chooses to ESCALATE rather than hard-code an answer when faced with unanswerable questions.
The benchmark consists of 200 manually constructed task items in four formats—route, classify, judge, and ground—of which one-fifth of the questions can only be answered correctly with ESCALATE.
The benchmark produces two numbers for each model that are never merged: a task score and a false-confidence rate (the proportion of cases that should have been escalated but were answered anyway).
Of the 12 hosted models measured, 7 maintained a false-confidence rate of 0.00 or 0.10 across every format.
Whether for small models or frontier large models, the worst-performing formats are consistently ground and classify, with no model showing its weakest performance on route or judge.
External reviewers identified three problems with the chain of custody: the chain could be severed or recalculated without anyone noticing, hashes retained only 64 bits, and a single line of garbled text caused a startup crash; all three have been fixed.
All Facts (6)
Developer Sean Campbell designed an evaluation framework called the Escalation Benchmark, which measures whether a model chooses to ESCALATE rather than hard-code an answer when faced with unanswerable questions.
50%UnverifiedThe benchmark consists of 200 manually constructed task items in four formats—route, classify, judge, and ground—of which one-fifth of the questions can only be answered correctly with ESCALATE.
50%UnverifiedThe benchmark produces two numbers for each model that are never merged: a task score and a false-confidence rate (the proportion of cases that should have been escalated but were answered anyway).
50%UnverifiedOf the 12 hosted models measured, 7 maintained a false-confidence rate of 0.00 or 0.10 across every format.
50%UnverifiedWhether for small models or frontier large models, the worst-performing formats are consistently ground and classify, with no model showing its weakest performance on route or judge.
50%UnverifiedExternal reviewers identified three problems with the chain of custody: the chain could be severed or recalculated without anyone noticing, hashes retained only 64 bits, and a single line of garbled text caused a startup crash; all three have been fixed.
50%