The Evaluation Trap: Your First Leaderboard Is Testing the Scorer, Not the Model

When designing a "Numeric Grounding" benchmark for a Kaggle Benchmarking Challenge, one developer uncovered a widely overlooked problem: the first leaderboard he published wasn't measuring model capability — it was measuring the flaws in his own scoring script. This seemingly minor lesson reveals a systemic risk in LLM evaluation that deserves serious attention: the scoring pipeline itself can become the largest source of error.

What the Benchmark Measures

This benchmark focuses on a specific yet critical capability: can a model answer numerical questions based solely on numbers stated in a short context, and choose to abstain when the number is absent? In financial or operational summary scenarios, fabricating a number is far more dangerous than honestly saying "I can't determine that."

To prevent models from relying on memorized knowledge, all companies, numbers, and units in the benchmark are fictional (e.g., made-up units like "Vendo Bin A" and "brens per lune"). The test has two layers, and each answer row on the final ANSWER line is scored 0 or 1:

• Sanity gate: Five task types, with 6 to 9 rows each, each isolating a classic grounding failure mode — reading a stated value (lookup), abstaining when absent (abstain), selecting the correct value from similar ones (distractor), multi-step arithmetic reasoning (multistep), and unit conversion using context-defined made-up units (units).
• Stress test ng-hard: 10 rows, each stacking multiple traps within a single paragraph — such as numbers revised twice (only the last revision counts), revisions that apply only to adjacent entities or years, approximate missing data (the question asks about "calendar 2024" while the context only provides "fiscal 2024"), and two-step unit conversion chains with three or more distractor numbers.

Worth noting is the author's methodological stance: he treated both the "scorer" and the "answer key" as suspects rather than trusting them by default. The answer key for every hardest task had to pass blind-vote validation from three models that weren't involved in scoring.

Numeric Grounding refers to a model's ability to strictly anchor its answers to numbers explicitly stated in a given context when responding to numerical questions, rather than relying on parametric memory or inferential gap-filling. This capability is critical in scenarios like RAG (Retrieval-Augmented Generation), financial analysis, and compliance review — a model that "hallucinates" industry averages or commonsense figures it saw during training can cause far more serious decision-making errors than simply refusing to answer. Unlike general QA, numeric grounding tasks require two simultaneous abilities: accurately reading values when the context supports them, and proactively abstaining when it doesn't. Mainstream benchmarks like MMLU and GSM8K focus primarily on the correctness of reasoning chains and offer relatively weak coverage of "restraint from fabrication" — which is precisely the angle this benchmark is designed to address.

Models Tested

The author selected models from four different vendors: Google's Gemini 3.7 Flash, Z.ai's GLM-5, Alibaba's Qwen 3 235B A22B Instruct, and OpenAI's gpt-oss-120b, all run on Kaggle.

The cross-vendor selection was intentional: if a parsing issue were specific to one model family's output format, it would show up as a score gap between families rather than a universal pattern. These four models also span a wide price range — a factor that turned out to matter more than the author had anticipated.

Four Key Findings

Finding 1: The First Leaderboard Was Testing the Scorer

The initial run produced what appeared to be a clear score distribution: 0.98, 0.94, 0.78, 0.62. But before publishing, the author replayed the models' raw responses and found that most answers marked "wrong" were actually correct.

The problem: the scorer only accepted lines beginning with ANSWER:. As a result, all of the following correct responses were scored 0:

• 27 ANSWER: 27 (number and label on the same line) — causing Qwen to score 0% on the distractor task
• **ANSWER: 72** (Markdown bold) — causing gpt-oss to score only 16.7% on multistep
• \text{ANSWER: } 135 (LaTeX format)

There was also an answer key error in one abstain row: the context stated "no side reactions occurred," and the question asked how many byproducts were generated. The correct answer is zero, but the author's key required "CANNOT BE DETERMINED."

Scorers/graders in LLM benchmarks play the critical role of converting free-text model outputs into comparable scores, and their robustness directly determines the validity of the entire evaluation. Common scoring approaches include: rule-based string matching (as used initially here), regex extraction, and using another LLM as a "judge model" (LLM-as-judge). String matching is simple and cheap but extremely sensitive to formatting — Markdown bold, LaTeX rendering, prefixed numbers, and any other formatting variation can cause missed detections. LLM-as-judge is more fault-tolerant but introduces the judge model's own biases and costs. The author opted for an improved rule-based extraction (scanning for the last ANSWER token) validated against a 21-format test table — a pragmatic middle ground that is more robust than pure string matching while remaining auditable.

Finding 2: After Fair Scoring, All Models Passed the Sanity Gate

The corrected scorer reads the last ANSWER token anywhere in the response; for numeric answers it takes the first number after the token, and for abstentions it requires the complete phrase CANNOT BE DETERMINED. Before going live, the author validated it against a table of 21 response formats, 8 of which must still score 0.

The parsing flaws plus that one bad key together shifted one model's score by as much as 0.38 across five core tasks (gpt-oss went from 0.62 to 1.00). After correction, all four models scored 100% on all five sanity gate tasks. In other words, in short, clean contexts, the sanity gate is a pass/fail baseline — not a ranking tool. Any model worth trusting with numbers should clear it easily.

Finding 3: The Stress Test Is Where Differences Emerge

Before running ng-hard, the author had three models not involved in scoring blindly check all 12 draft answer keys, retaining only those at least two models agreed on. Two rows were removed. In one of them, all three checkers answered 6.40 for "North Ridge," while the context only stated "North Ridge Annex" — this near-match trap fooled strong models, and a row the author himself couldn't defend was no better than a coin flip with extra steps.

Gemini 3.7 Flash, GLM-5, and Qwen 3 235B all scored 100% on ng-hard; gpt-oss-120b scored 80% (8 out of 10 rows), with two distinct failure types:

• Arithmetic error: The context gives a rate revised twice to a final value of 11.60 brens/lune, plus a run of 3.0 lunes and a deduction of 1.20 brens. The correct answer is 11.60×3.0+1.20=36.00, but gpt-oss answered 4.20 — exactly 3.0+1.20, completely ignoring the rate.
• Incomplete abstention: The question asks about actual 2025 capacity; the context only provides 2025 planned capacity and 2024 actual capacity. The entire visible response from gpt-oss was ANSWER: CANNOT BE — correct judgment, but missing one word. Scoring this row 0 is a judgment call that could go either way.

So gpt-oss's total score of 0.97 (versus 1.00 for the other three) rests on one clear error and one incomplete response.

The "blind-vote answer key validation" step corresponds to the inter-annotator agreement principle in academic evaluation. Traditional dataset construction typically requires multiple human annotators to independently label the same sample, then calculates agreement metrics like Cohen's Kappa or Fleiss' Kappa — samples below a threshold are discarded or re-discussed. This article transfers that approach to the LLM era: models not participating in the competition scoring serve as "independent annotators" who blindly review each answer key, with majority agreement as the retention criterion. This approach is both economical (no human labor required) and effective at filtering out rows that even strong models can't answer consistently — questions that are fundamentally ambiguous. The deleted "North Ridge vs North Ridge Annex" row illustrates how near-match traps, when phrasing is even slightly ambiguous, can end up testing the ambiguity of the question itself rather than model capability.

Finding 4: Same Perfect Score, Wildly Different Bills

Kaggle's "score vs. total cost" chart across the full six-task run shows:

• Qwen 3 235B: 100 points, $0.006
• Gemini 3.7 Flash: 100 points, $0.07
• GLM-5: 100 points, $0.19
• gpt-oss-120b: 96.67 points, $0.005

Among the three perfect-scoring models, GLM-5 cost roughly 30× more than Qwen. Once accuracy is tied, the cost per flawless run becomes the genuinely useful tiebreaker.

Implications for Evaluation Practice

This small-scale experiment (only 6 to 10 rows per task, a single run per model, with results that are only indicative) yields lessons far more valuable than the scores themselves:

1. Replay raw responses before trusting the leaderboard. Most of the score differences the author initially measured came from parsing, not the models.
2. Validate answer keys independently. Blind-vote checking with models not involved in scoring caught keys the author couldn't defend himself.
3. Single-trap tests are now just a pass/fail bar for these four models; stacked traps are what create separation.
4. When accuracy ties, cost decides — a ~30× cost difference between three perfect-scoring models.

The author also plans further validation: placing the same traps inside long documents and tables, and running each model multiple times to determine whether gpt-oss's incomplete abstention recurs or was a one-off.

For any team building their own evaluation suite, the core takeaway from this practical account is straightforward: you think you're measuring the model, but until you've proven that the scoring pipeline is trustworthy, you might just be measuring your own bugs.