LLM Judge Score Drift: Why Nightly Eval Gates Give False Alarms

Title

LLM Judge Score Drift: Why Nightly Eval Gates Give False Alarms

Meta Description

Nightly LLM eval gate false alarms often stem from judge score drift. This article analyzes how the typed judge JEV uses typed verdicts and calibrated probabilities to improve repeatability, and offers selection guidance for four check types—format, semantic, tone, and high-volume—plus adversarial input protection strategies.

Article Content

A customer service Agent maintained by a developer runs automated evaluations every night against ten thousand historical tickets. One morning, the results turned red. No Prompt changes had been merged, the model version was pinned, and the retrieval index had not been rebuilt. After finding no configuration differences, the answer appeared in the judge logs: for the same ticket and the same reply, the score was 0.7 the previous night and 0.6 that night, while the pass line was exactly 0.7. The Agent was identical on both nights, but the judge gave different scores for the same input.

This is not an isolated case. LLM judges produce slightly different scores on every run, and when a gate evaluates thousands of cases, some red alerts are actually the judge disagreeing with itself.

Regression Gates Need Repeatable Verdicts

LLM-as-a-judge performs excellently when judging complex outputs: give a strong model a rubric, and it returns a score with a sentence of explanation. That explanation is extremely useful when debugging a single Bad Case—you can read it, challenge it, and record it.

But a gate requires something else: for the same input, it must return the same verdict every time, so that a failing case means the Agent has truly regressed. Code checks and semantic validators naturally satisfy this requirement and can independently support a gate. LLM judges need workarounds, such as averaging multiple judge runs for important cases, but this multiplies the cost of the slowest and most expensive checks.

Where LLM judge variance comes from. Even if the inference provider sets the temperature parameter to 0, outputs under multiple concurrent paths may still have subtle differences—this is not a Bug, but the norm of current inference stacks. One cause is non-determinism in floating-point operations: GPU parallel reductions, operator fusion order, and different batch sizes can all cause the same input to follow different accumulation paths. Another cause is batch effects: other cases the judge sees in the same batch can indirectly influence the log probability distribution of the current case through attention mechanisms, so running alone and running in sequence may produce different scores. A third cause is server-side version drift: although weights are pinned, inference kernels, quantization methods, and context chunking strategies can all be silently updated on the vendor side, meaning the same API string does not call the same computation. Together, these three produce the phenomenon of "model version pinned, index not rebuilt, yet scores changed." To diagnose it, run the same batch of inputs five to ten times consecutively and observe whether the score distribution overlaps on both sides of the pass line—if it overlaps, that threshold cannot distinguish real regressions from noise.

Trade-offs of Typed Judges

According to the paper "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure," JEV is a typed evaluator released by TypeSafe AI, positioned as a fast, intuitive-scoring "System One" model. Its most notable difference is the output: JEV returns a typed verdict (yes/no, multiple choice, or a point on an ordered scale) with a calibrated probability, and writes no reasoning process at all.

Paper data shows that JEV has a median latency of 0.15 seconds and costs only 0.36% of the strongest comparison model; when the verdict can be read directly from the given text, the gap with GPT-6 is within three percentage points. But on tasks requiring derivation, such as math, code, and logic, the gap widens. These data are based on the paper's benchmark setup; actual traffic performance must be measured independently.

JEV trades written reasoning for repeatability. For a regression suite that runs every night, this trade-off is worth it.

What a "typed" evaluator is. "Typed" means the output falls within a predefined set of values: boolean, fixed options, or a point on a discrete scale, rather than a piece of free text. This brings two engineering benefits. First, the verdict can be consumed directly by programs without parsing natural language—parsing itself is a potential source of error. Second, a limited value space means a probability distribution is available, and the model can simultaneously give "choose this option" and "how confident it is in choosing this option," i.e., a calibrated probability. Calibration means that when the model claims 80% confidence, over the long run about 80% of those claims are correct; an uncalibrated score is only a relative magnitude and cannot be used as a threshold. A typed judge compresses this mechanism into a single forward pass: no reasoning chain, no autoregressive decoding, so latency can be as low as sub-second and cost scales linearly and comparably with call volume. The cost is that it cannot handle tasks requiring multi-step derivation, because the answers to those tasks are not in the given text but after deriving from the text.

Probability Is the Most Frequently Used Part

Calibrated probability changes how a judge is integrated: accept the verdict directly at high confidence, and route to a stronger reasoning judge or human handling at low confidence, making the typed judge a triage step before expensive checks.

Thresholds must be set independently and first validated on your own labeled cases, because copying cutoff values from someone else's benchmark only describes their data.

One Agent, Four Checks, Four Methods

Choose the method based on what is being checked. For a customer service Agent, there are four categories:

Format checks: whether the reply is valid JSON with required fields. Code checks handle this—deterministic, instant, and free.

Reference consistency: whether the reply matches the known solution for that ticket. A semantic validator handles this at low cost. Embed the reply and the reference, then take cosine similarity; above the threshold passes. Lightweight models such as all-MiniLM-L6-v2 can run locally. Start tuning the threshold from 0.75; too low lets paraphrases of wrong answers through, too high misses correct answers with different wording.

Tone judgment: politeness and brand voice are subjective judgments, and this is where LLM judges prove their value. Use pointwise scoring, comparing a single answer against a threshold. When ranking is needed, switch to pairwise comparison and run both orders, because models tend to favor the answer they read first.

High-volume gates: the high-throughput gate over ten thousand historical tickets every night is exactly where typed verdicts shine.

Why the four methods cannot be merged into one. Each check has a different "answer space," which determines what implementation it is naturally suited to. The answer to a format check is a predefined valid/invalid, a finite-state verdict, and code completes it in nanoseconds with absolute reproducibility. The answer to reference consistency is "whether the semantics are equivalent," essentially a distance comparison in continuous space, and the semantic validator reduces it to a scalar: cosine similarity. The answer to tone judgment is "whether it meets a subjective standard," with no precisely writable rule, and it can only be approximated by a model that understands natural language. The answer to a high-volume gate is "which items in this batch regressed," requiring a reproducible pass or fail for each of ten thousand items under a fixed budget. Stuffing these four into the same LLM judge is equivalent to making a component good at subjective judgment take on the deterministic tasks it is worst at, wasting cost and introducing variance. Conversely, handing tone judgment to regular expressions misses every reply that is genuinely polite but phrased differently.

Scenarios That Still Need Reasoning Segments and Human Intervention

Two types of scenarios retain a reasoning judge or human intervention:

Derivation tasks. When the verdict depends on checking math, code, or multi-step argumentation, the gap of typed judges widens, and such cases escalate by default.

Adversarial inputs. Both typed and generative judges read the content being scored, and that content may carry instructions aimed at the judge. Written reasoning cannot reliably expose Prompt injection, and typed verdicts leave less information to investigate afterward. When a judge participates in safety or moderation decisions, treat its input as untrusted, isolate evaluation instructions, test adversarial samples, use deterministic rules where policy can be written precisely, and route uncertain cases to human review.

Advice to My Past Self Before the Morning of the Red Alert

• Using a judge with variance as a gate: a red build that cannot be reproduced may be judge variance; use low-variance checks for gates.
• Treating similarity as truth: "Paris is the capital of France" and "Paris is not the capital of France" have extremely high similarity. Scenarios where flipping one word changes the verdict need to be paired with rules or a judge.
• Borrowing confidence thresholds: calibrate on your own labels first.
• Letting one judge evaluate everything: format, reference, tone, and volume each have suitable methods, and most production Agents need three to four in parallel.

The full comparison, decision table, and Python examples for semantic validator and LLM judge patterns can be found in the original link.

Which check in your suite turns red when nothing has changed? For this developer, it was the nightly regression gate, and the fix was to hand it to a check that gives nearly the same answer every time.

One-line Summary

LLM judges have score drift, so gates should use typed low-variance checks and retain reasoning judges for difficult cases.

Paragraph Summary

The article starts with a real failure: a customer service Agent's nightly evaluation suddenly turned red, but the Agent itself was identical on both nights—what changed was the score an LLM judge gave for the same input. The same reply scored 0.7 the previous night and 0.6 that night, just crossing the 0.7 pass line. The author points out that LLM-as-a-judge is good for debugging complex outputs but unsuitable as a regression gate requiring repeatable verdicts, because every run introduces slight variance that creates many false alarms across thousands of cases. The paper's JEV typed judge replaces written reasoning with typed verdicts plus calibrated probabilities, trading for a median latency of 0.15 seconds and extremely low cost. The author recommends choosing methods by check content: code for format, semantic validators for reference consistency, LLM judges for tone, typed verdicts for high-volume gates, while derivation tasks and adversarial inputs still need reasoning judges or human intervention.