Sean Campbell
An AI developer and the designer of the Escalation Benchmark, focused on researching metacognitive capabilities and calibration evaluation in AI models.
Timeline (last 90 days)
Developer Sean Campbell designed an evaluation framework called the Escalation Benchmark, which measures whether a model chooses to ESCALATE rather than hard-code an answer when faced with unanswerable questions.
The author recommends ranking models by their worst format (floor) rather than by their average.
The author tested 5 local models ranging from 1B to 8B parameters on their own machine at temperature zero, and every single one produced a 100% false-confidence rate on at least one entire format.
The author's methodological summary is: rank models by their floor rather than their average, build systems that do not require models to be self-aware and use code to verify every statement they make, and score yourself with the same benchmark.
All Facts (4)
Developer Sean Campbell designed an evaluation framework called the Escalation Benchmark, which measures whether a model chooses to ESCALATE rather than hard-code an answer when faced with unanswerable questions.
50%UnverifiedThe author recommends ranking models by their worst format (floor) rather than by their average.
50%UnverifiedThe author tested 5 local models ranging from 1B to 8B parameters on their own machine at temperature zero, and every single one produced a 100% false-confidence rate on at least one entire format.
50%UnverifiedThe author's methodological summary is: rank models by their floor rather than their average, build systems that do not require models to be self-aware and use code to verify every statement they make, and score yourself with the same benchmark.
50%