Why Do LLM Judge Results Keep Flip-Flopping? Breaking Down the Stability Traps of LLM-as-Judge

The Problem: Same Input, Scores That Keep Jumping

A developer posted on Reddit about a pain point that many LLM-as-Judge practitioners will recognize: using an LLM to evaluate text records against a checklist. Each record contains several free-text fields, and the model scores roughly 10 yes/no criteria in a single prompt — a record is only marked "complete" if every criterion passes.

The catch: re-running the exact same dataset produced different totals — 11/27 passing the first time, 12/27 the second time, with zero changes to the input. Whenever one criterion flipped on one record, the top-line metric shifted with it. This case is practically a textbook example of the "non-reproducibility" problem in LLM-based evaluation.

Patterns Discovered Through Practice

The original poster had already done considerable debugging, and the observations themselves are worth examining:

Temperature and random seed help, but don't fix the root cause. Setting temperature to 0 and fixing the seed did improve things somewhat, but results still flipped. The reason: many inference backends treat seed as "best effort" — floating-point operation ordering, batch scheduling, and hardware differences can all introduce non-determinism. In other words, temperature=0 does not guarantee deterministic output.

Vague questions are essentially coin flips. Subjective judgments like "Is this content comprehensive?" are nearly random, while factual checks like "Does X exist?" rarely flip. This reveals a core tension: the objectivity of the evaluation criteria determines the stability of the results.

Models take examples hyper-literally. The author listed "word A or B" as an example of vague phrasing, and the model then passed any text that avoided those exact two words. Examples in a prompt are treated as hard rules, not semantic references.

The Cascade Effects of Prompt Wording

Even subtler was how wording details produced unexpected consequences:

• Removing "does it" from a question turned it into an instruction, causing the model to generate its own answer and then mark nearly everything as passing.
• Rewriting one criterion caused pass rates for other criteria in the same prompt to shift by 5–10 percentage points.
• Even when the prompt explicitly said "only look at field B," the model still pulled evidence from field A roughly one-third of the time.

These phenomena show that multiple tasks packed into a single prompt interfere with each other — the model isn't evaluating each criterion independently, but making holistic judgments within a shared attention context.

---

The relationship between temperature and determinism deserves a closer look. The temperature parameter controls how "flat" the probability distribution is during sampling: setting it to 0 means always selecting the highest-probability token (greedy decoding), which should theoretically produce identical outputs. In practice, however, inference systems have multiple sources of non-determinism: the order of parallel reductions in GPU floating-point operations, non-deterministic CUDA kernels, dynamic batching (which merges multiple requests into a single computation and alters numerical precision), and some providers routing requests to different hardware nodes for load balancing. The seed parameter only fixes the random number generator on the model side — it can't control any of these lower-level factors. This is why major providers like OpenAI and Anthropic explicitly document seed's determinism guarantee as "best effort" rather than strict: even with identical parameters, outputs across calls can still diverge at tokens with borderline probabilities.

---

Why "All Must Pass" Amplifies Noise

The observation the original poster made at the end is worth remembering for anyone building evaluation pipelines: "Ten slightly noisy checks stacked together produce an extremely noisy result."

This is a probability problem. Suppose each criterion has a 95% accuracy rate on its own — already quite high. But under an all-must-pass logic, the probability that a record is judged correctly overall is approximately 0.95 to the power of 10, which is only about 60%. Noise is multiplied when components are chained in series, and "completion rate" is exactly the most fragile form of aggregation.

This also explains why the author saw flips happening at the margins: whenever one or two records oscillate between 9/10 and 10/10, the total count jumps.

---

This phenomenon has a direct parallel in reliability engineering: series system reliability. When n independent components must all function correctly, the system's overall reliability equals the product of each component's individual reliability — the same principle behind why "a test suite's fragility grows exponentially with the number of test cases" in software testing. In LLM evaluation, the problem is further amplified because the criteria aren't truly independent (they share the same context window and are coupled at the attention level), meaning actual noise is typically higher than what the independence assumption would predict. Statistically, these evaluation results are better expressed as confidence intervals rather than point estimates: reporting "pass rate is between 38%–52% (95% CI)" is far more honest than reporting "11/27," and far more useful for downstream decision-makers assessing whether the result is credible.

---

Practical Improvement Strategies

Based on the observations in the thread, there are several approaches worth trying:

Split, don't merge. Break 10 criteria into separate prompt calls to eliminate cross-criterion interference. More expensive, but significantly reduces cross-contamination.

Keep only objectively verifiable criteria. Redefine subjective questions like "Is it comprehensive?" as quantifiable factual checks, or flag them explicitly as low-confidence dimensions that shouldn't drag down the overall verdict.

Use multiple samples with majority voting. Since single-call outputs are unstable, run 3–5 calls on the same record and use majority voting to determine the result. This is one of the most practical engineering tools for combating LLM evaluation noise.

Redesign the aggregation logic. "All must pass" is too strict and too fragile. Consider weighted scoring, allowing a small number of failures, or reporting confidence intervals rather than a hard completion count.

Give examples semantic context, not just literals. Explain in the prompt that examples are "representative of this type of question," provide both positive and negative examples, and avoid letting the model treat example words as the sole decision criteria.

---

A few engineering details worth noting for majority voting implementation. First, use an odd number of samples (3, 5, 7) to avoid ties. Second, each call should be made independently (not reusing the same session context) to ensure sampling independence. A more sophisticated approach is self-consistency prompting: have the model output a chain-of-thought before rendering its judgment, then aggregate across multiple reasoning paths — research shows this is more stable than direct majority voting on borderline cases. For cost-sensitive scenarios, consider a cascade strategy: use a single call to filter out clearly passing or clearly failing cases with high confidence, and only run multiple samples on records that fall in the uncertain zone — a practical balance between accuracy and cost.

---

Closing Thoughts

The value of this case study is that it's not a bug — it's an inherent characteristic of LLMs as evaluation tools: language model outputs are fundamentally probability distributions, while evaluation tasks expect deterministic boolean results. The tension between these two things can't be resolved by tweaking a single parameter. It can only be systematically suppressed through task decomposition, repeated sampling, objective criterion design, and robust aggregation logic.

For any team building automated evaluation pipelines, this is a sobering reminder: before trusting the numbers from an LLM judge, first test how reproducible it is on the same input.