Auditing LLM Judges: Evaluation Failures and Fixes in Text-to-SQL Pipelines

LLM-as-Judge: The Unvalidated Referee

In production text-to-SQL pipelines, a increasingly common practice is to use large language models as "judges" (LLM-as-judge) to assess the faithfulness of generated SQL outputs. The problem is that the agreement between these judges and human annotators is rarely, if ever, actually measured.

A study published on arXiv ("Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline") confronts this blind spot head-on. The researchers audited their own deployed gpt-4o-mini judge, and the results are sobering: on a disagreement-enriched dataset, the judge's Cohen's kappa agreement with a gold standard confirmed by two authors was just 0.04 — nearly indistinguishable from random guessing. Even on a uniformly random sample, kappa was only 0.42.

More alarming still, this judge over-flagged 77.1% of human-confirmed FAITHFUL cases on the enriched set — meaning it incorrectly raised red flags on a large proportion of correct outputs, making its reliability deeply questionable.

Cohen's kappa is a statistical measure of agreement between two evaluators, ranging from -1 to 1. Its key advantage is that it discounts agreement due to pure chance: kappa = 0 means the level of agreement is no better than random, kappa = 1 indicates perfect agreement, and values between 0.6–0.8 are generally considered "strong agreement." In evaluation tasks, simply computing accuracy inflates results (since even random guessing achieves some hits), making kappa a stricter benchmark for assessing annotator or judge reliability. The study's juxtaposition of the gpt-4o-mini judge's kappa = 0.04 against random guessing's kappa = 0 is a deliberate emphasis: this judge provides virtually no judgment capability beyond chance.

GRADE-HALLUCINATION: The Core Failure Mechanism

The researchers traced these misfires back to their root cause and found that the vast majority of over-flagging pointed to a single mechanism, which they named GRADE-HALLUCINATION.

This finding is illuminating: the judge's errors are not scattered random noise, but a systematic, attributable, single failure mode. That means evaluation breakdowns can be diagnosed, localized, and specifically repaired — rather than requiring a blind swap of the entire model. For any team relying on LLMs for automated evaluation, identifying this kind of structured failure mechanism is a prerequisite for building trustworthy evaluation pipelines.

GRADE-HALLUCINATION refers to a specific hallucination pattern that emerges when a judge model scores SQL query results for faithfulness: rather than reasoning about actual query semantics or output correctness, the judge develops erroneous sensitivity to surface-level features — such as numeric formatting, column naming conventions, or result ordering — and incorrectly flags genuinely correct outputs as unfaithful. This naming distinguishes the judge's failure from standard LLM generation hallucinations: the problem doesn't arise at the SQL generation stage, but within the evaluation stage itself. Recognizing this as a structured, single failure mode (rather than random noise) means it can be systematically eliminated through targeted prompt engineering, judge fine-tuning, or output post-processing — without blindly replacing the entire evaluation framework.

Swapping the Judge: Smaller Models Can Match Flagship Performance

To address the failing judge, the research team tested alternatives. A self-hosted Qwen3.6-27B model used as a replacement judge achieved a kappa of 0.72, putting it in the same range as Claude Opus 4.7 (kappa = 0.71).

The researchers honestly note that this head-to-head comparison, conducted on n=96 samples, is statistically underpowered. But from a practical deployment decision standpoint, this barely matters — because Qwen costs approximately 1/300th of Claude Opus per call. When performance is comparable but cost differs by two orders of magnitude, the choice is obvious.

This offers a pragmatic signal for the industry: for evaluation tasks, self-hosted mid-sized open-source models may provide an extremely cost-effective option, without needing to pay for the most expensive flagship APIs.

Ensembling Is Not a Free Lunch

An intuitively appealing approach is to "ensemble" multiple judges to improve reliability. The research results offer a cautionary note: ensembling is not inherently beneficial.

Pairing a weak judge with a strong one actually decreases overall agreement — the weak model drags down the strong one. By contrast, having three strong judges collaborate under a unanimity routing scheme achieves kappa = 0.79 while maintaining 89.7% automatic coverage.

This reminds engineering teams that the gains from ensembling depend on member quality and routing mechanism design. Simply stacking models not only fails to improve results — it can actively introduce negative interference.

Cross-Domain Auditing: Even Expert Gold Standards Are Unreliable

One of the study's most thought-provoking findings comes from applying the same audit methodology to out-of-domain datasets. When the audit pipeline was applied to BIRD-financial — a widely used benchmark — it flagged 25.5% of expert-written gold standard SQL queries as having candidate quality issues.

This figure points to a deep vulnerability in the entire text-to-SQL evaluation ecosystem: even data annotated by experts and treated as a "gold standard" may contain a substantial proportion of problematic samples. If the benchmark itself is noisy, then the model rankings and performance claims built on top of it all warrant reexamination.

The research team open-sourced their code and pre-registration on GitHub (github.com/JamesL404/synca-audit), making this audit recipe reproducible and transferable to other production pipelines.

BIRD-financial is the financial subset of the BIRD (Big Bench for Large-Scale Database Grounded Text-to-SQL Evaluation) benchmark, with SQL answers written by human domain experts. It is widely used by both academia and industry to evaluate text-to-SQL model performance. Because its data is annotated by domain experts, it has long been assumed to be a high-quality "gold standard" reference. When this study applied its audit tools to this broadly trusted dataset and found quality issues in roughly one quarter of entries, it has direct implications for teams using BIRD leaderboard scores as a model selection criterion — top-ranked models may have partially succeeded by "learning to adapt" to the noise in the benchmark, rather than by genuinely mastering more accurate SQL generation.

Implications for Practitioners

The value of this work lies not only in exposing the problems of a single deployment, but in providing an actionable methodology:

• Measure before you trust: Any LLM judge should have its agreement measured against human gold-standard data before going into production — don't assume reliability by default.
• Diagnose failure mechanisms: Systematic errors like GRADE-HALLUCINATION, once localized, can be specifically repaired.
• Balance cost and performance: Self-hosted open-source models can match flagship-level performance at a fraction of the cost.
• Treat benchmark data with skepticism: Even expert gold standards may have a 25% problem rate — evaluation conclusions should be held with appropriate caution.

For every team currently relying on LLMs for automated evaluation, this research is a timely reminder: the evaluator itself also needs to be evaluated.