When Should an AI Model Say "I Don't Know"? Findings from a Kaggle Escalation Benchmark

A deceptively simple question runs through this eleven-day benchmark: does your model know when it doesn't know? Developer Sean Campbell designed an evaluation framework called the "Escalation Benchmark" to measure whether models honestly choose ESCALATE (hand off / defer) when faced with unanswerable questions — rather than fabricating a response. The conclusions are thought-provoking, and challenge the way we typically evaluate models.

Benchmark Design: Two Numbers That Never Merge

The test comprises 200 hand-crafted tasks spanning four shapes: route, classify, judge, and ground. Crucially, one fifth of the questions can only be answered correctly with ESCALATE — they are fundamentally unanswerable, and the right move is to admit it and escalate.

The benchmark produces two numbers for each model, and the author insists they are never combined: a task score and a false-confidence rate — the proportion of times a model should have escalated but answered anyway. This deliberate split is the benchmark's core value proposition: it refuses to hide a model's fatal weakness behind a clean composite score.

The "false-confidence rate" metric draws on a long-standing concern in AI safety around calibration. A well-calibrated model is one whose expressed confidence matches its actual accuracy — if a model says "I'm 90% sure," it should be right roughly 90% of the time in those cases. Yet mainstream leaderboards (like MMLU, HumanEval, etc.) typically measure only accuracy without penalizing overconfidence, which encourages models during training to treat any answer as better than silence. The ESCALATE mechanism essentially turns "I don't know" into a scorable, valid output rather than a missing value. This design is especially critical in high-stakes domains like customer service bots, medical Q&A, and legal assistance — a model that scores high on accuracy but operates with 100% false confidence is far more dangerous than a low-scoring model that escalates frequently.

What the Tests Found

The author summarized three key findings, explicitly emphasizing that "none of this is a leaderboard."

Averages Hide the Floor

The sharpest observation: rank models by their worst shape, not their average. Take Claude Haiku 4.5 — its overall performance lands around the one-third mark, which looks fine. But on the judge shape, it answered 9 out of 10 questions that should have been escalated — a 90% false-confidence rate. Looking at averages alone completely buries this fatal flaw. The author's recommendation is therefore unambiguous: rank by the floor, not the mean.

Most Models Do Stop

The good news is that "just guessing" isn't universal. Of the 12 hosted models tested, 7 maintained a 0.00 or 0.10 false-confidence rate across every shape. In other words, the "bluffers" who force answers are actually the minority — and averages are precisely where they hide.

Ground and Classify Are Always the Hardest

Whether nano-scale tiny models or frontier large models, the worst-performing shapes were consistently ground and classify — not a single model had its weakest point in route or judge. This cross-scale consistency suggests these two task types inherently demand more of the "knowing what you don't know" capability.

Local Small Models: Can Do the Task, Don't Know When They Can't

The original motivation behind this research pointed at a concrete question: can locally-run models on a laptop recognize their own limits the way frontier models can? The answer is no — decisively so.

The author tested 5 local models ranging from 1B to 8B parameters at temperature zero on a local machine. Every single one answered confidently through at least one entire shape — delivering 100% false-confidence rates precisely where ESCALATE was the only correct answer. The best performer, llama3.1:8b, still guessed on 31 of 40 unanswerable questions. Every frontier model on Kaggle stayed at 5% or below.

In one sentence: small models can handle tasks reasonably well, but are almost completely unaware of when they can't. This is exactly the author's argument for the entire system — these models cannot be deployed in production alone; there must be a "chain" above them to catch failures.

The failure of local small models at metacognition is closely tied to their parameter scale and training approach. Large frontier models typically undergo RLHF (Reinforcement Learning from Human Feedback) or variants of alignment training that explicitly include demonstrations of "refusing to answer" or "expressing uncertainty," teaching the model to emit uncertainty signals in specific contexts. Smaller 1B–8B models, constrained by parameter capacity, are often distilled or fine-tuned primarily to maximize task completion rates. The ability to "know what you don't know" tends to rank lower in priority during compression and is easily lost. Furthermore, the temperature-zero inference setting the author used further suppresses the probability of outputting ESCALATE — under greedy decoding, models favor the response patterns most frequent in training data, and "escalate" is inherently rare in instruction-tuning datasets.

Four Day-0 Predictions: How Did They Hold Up?

On Day 0, before running a single test, the author wrote down four predictions to grade himself against later. This kind of self-imposed constraint is itself worth admiring:

• At least one frontier model exceeds 20% false-confidence rate (Hit): Claude Haiku 4.5 hit 90% on judge. Though the author admits this depends on whether Haiku counts as "frontier" — all four top-tier models were under 5%.
• The best local model ≤4B beats at least one frontier model on false confidence (Miss): Not even close. The best small model still bluffed on 80% of unanswerable questions.
• Task score and false confidence are only weakly correlated, Spearman below 0.5 (Miss): The actual result was a strong correlation — Spearman of −0.79. The better a model performs on tasks, the less it bluffs. They are not independent.
• Submissions land on time, no more than one day late (Hit).

The two missed predictions are actually the most informative. Especially the third — task capability and "knowing what you don't know" turned out to be strongly correlated, the opposite of the author's original intuition.

From Benchmark to System: Let Code Be the Safety Net

The most engineering-valuable step is when the author "inverts" the benchmark and embeds it into an actual system. Rather than continuing to ask models "do you know when you don't know," he simply removes that requirement.

The concrete mechanism works like this: a terminal acts as a "seat," and code constructs and prints a set of "boxes" in a fully deterministic way, then has the local small model qwen3:4b produce one sentence about them. The code then verifies that sentence — every reference must be in the pool, every quote must be exact, every number must come from a box, no fabrications allowed. If the sentence fails validation, or the model simply doesn't respond, the system discards it and uses a plain-text sentence written by the code itself.

This is ESCALATE built into the system: the code knows what's in the boxes, and anything not in them gets handed straight back. The model no longer needs to be self-aware, because that judgment has been delegated to deterministic code.

This architectural pattern of "code validates model output" is commonly known in engineering practice as guardrails or structured output validation. The core idea is to treat the language model as an untrusted draft generator rather than a final decision-maker, with a deterministic logic layer responsible for fact-checking. Related engineering concepts include hallucination detection and RAG (Retrieval-Augmented Generation) — the latter architecturally shrinks the space for hallucination by requiring the model to cite only retrieved document chunks. The validation logic the author describes — "every reference must be in the pool, every quote must be exact" — is essentially a lightweight post-hoc RAG verification: rather than constraining model generation, it performs line-by-line comparison against a trusted data source after generation, falling back to deterministic text when validation fails. The engineering advantage of this approach is that it requires no model replacement — just a pure-logic filter layer added at the end of the inference pipeline.

The Benchmark as a Mirror for Your Own Code

Interestingly, this benchmark didn't just reveal problems in the models — it also exposed bugs in the author's own code. It turned out to be a mirror.

External review found three holes in the chain of records: the chain could be severed or recalculated without anyone noticing, hashes were only 64 bits, and a single garbled line would crash startup. All three were fixed — hashes upgraded to full digests, and garbled lines now produce a numbered "breakpoint" instead of a crash.

A review of willow-bot uncovered 20 instances of "code behavior not matching documentation" — the docs say one thing, the code does another; reads correctly, runs incorrectly. The author's favorite bug: Python's splitlines() splits not only on newline characters but also on Unicode line separators, so a single character inside a record could silently slice it in half. The fix was to split only on \n.

There was also a key verification problem: a seal was being compared against the wrong keyring — the machine's, not the signer's public key. The step that "decides whether something is true" had been verifying the wrong thing all along.

Three Pieces of Advice for Day-0 You

The author distilled the entire project's methodology into a few sentences worth keeping for engineering practice:

Don't rank models by their average — rank them by their floor. Don't expect models to know when they don't know; instead, build systems that don't require them to, and use code to verify every claim they make. Finally, use the same benchmark to grade yourself — it caught a score the author never actually gave, a scanner that never ran, a hook that failed to "fail closed," and 20 places where the bot didn't behave according to its own documentation.

"Every one of them read correctly. Not one of them actually was. It's a mirror, and that was the point from the start."