When LLM Judges Agree, Should We Actually Trust Them?

The Rise of LLM-as-a-Judge

Using large language models (LLMs) to evaluate the output quality of other models has become a mainstream practice in AI evaluation. Compared to the high cost and inefficiency of human annotation, the "LLM-as-a-judge" approach enables fast, large-scale scoring and comparison of model-generated text. When multiple LLM judges converge on the same verdict, people instinctively tend to view that judgment as more reliable — after all, "consensus" in the human world usually signals proximity to truth.

But does this intuition actually hold up? An article that sparked discussion on Hacker News raises a critical question: When LLM judges agree with one another, should we actually trust them?

Agreement ≠ Accuracy

The core issue is that agreement among multiple LLM judges may not stem from each independently converging on the correct answer — it may simply reflect the systematic biases they share.

Most mainstream large models today are trained on similar internet corpora, using comparable architectures and alignment methods (such as RLHF). This means they are likely to inherit the same biases, blind spots, and tendencies. For example, a well-documented phenomenon is "verbosity bias" — a tendency to rate longer, more detailed responses higher, even when those responses aren't necessarily more accurate. Models may also share preferences for particular writing styles, formats, or tones.

In this context, the agreement among several LLM judges is essentially the same bias expressed repeatedly — not a cross-validation of independent perspectives. Statistically, majority voting only reduces error rates when errors are mutually independent. When errors are highly correlated, "consensus" actually amplifies mistakes and creates a false sense of credibility.

RLHF (Reinforcement Learning from Human Feedback) sits at the core of mainstream LLM alignment: models generate multiple candidate responses, human annotators rank them, those preference signals are used to train a "reward model," and the language model is then fine-tuned via reinforcement learning to maximize the reward score. Because annotator pools across different organizations tend to share similar cultural backgrounds, education levels, and aesthetic preferences — and because reward models themselves crystallize these preferences into optimizable signals — LLMs from different vendors often converge on similar "people-pleasing" strategies: responses that are well-structured, affirmative in tone, and relatively lengthy. This explains why models from different companies, when acting as judges, can still exhibit highly correlated biases: their "tastes" have been shaped by fundamentally similar human curation processes.

The Hidden Trap of Correlated Errors

This phenomenon is especially dangerous in evaluation practice. Researchers and engineers often use inter-judge agreement as a proxy for quality, and sometimes embed it directly into automated pipelines. Once the errors across judges are correlated, the entire evaluation system can be systematically misled.

For instance, if all judges share a preference for a particular mode of expression, a genuinely superior answer that doesn't fit that "taste" will be consistently underrated, while a superficially polished but less substantive answer will be consistently overrated. Because the judges "speak with one voice," the problem is far harder to detect — agreement here masks bias rather than revealing truth.

The Condorcet Jury Theorem from statistics is often invoked to justify majority voting: if each juror makes independent judgments and has a better-than-50% accuracy rate, collective accuracy approaches 100% as the number of jurors grows. However, the theorem rests on a strict prerequisite — errors across jurors must be mutually independent. When judges share training data, architecture, or alignment procedures, this independence assumption simply does not hold, and the theorem's conclusions collapse with it. Worse, correlated errors are systematically amplified under majority voting: a case that all judges collectively misjudge will appear, in the vote tally, with exactly the same confidence profile as a case they all correctly and unanimously endorse — nearly indistinguishable from the outside.

How to Use LLM Judges More Carefully

None of this means LLM judges are worthless — it's a reminder to interpret their conclusions with appropriate skepticism. A few practical directions worth considering:

• Introduce heterogeneous judges: Use models from different vendors and training lineages to reduce the correlation between their errors.
• Retain human spot-checks: At critical evaluation checkpoints, use human judgment to sample and audit LLM consensus, surfacing systematic biases.
• Pay attention to disagreements, not just agreement: Points of divergence among judges are often more informative than points of consensus and deserve deeper analysis.
• Account for known biases: For documented biases like verbosity preference and positional bias, design controlled experiments to manage their influence.

"Positional bias" is another systematic bias documented across multiple studies: when asked to pick the better answer from two options, LLMs tend to favor whichever option appears first or last, rather than the one that is genuinely higher quality. This bias is independent of the content itself and stems purely from a language model's sensitivity to sequence position. The standard method for controlling it is "symmetric evaluation": evaluate both answers twice, swapping their positions, and only treat the result as valid if both evaluations reach the same conclusion. For verbosity bias, partial control can be achieved by imposing a fixed word-count ceiling or requiring judges to explain their reasoning alongside their scores — the latter helps expose whether a judge is truly responding to content quality or to surface-level features.

Closing Thoughts

The question "when LLM judges reach consensus" cuts to a foundational assumption in automated AI evaluation. The confidence that consensus inspires may be nothing more than a statistical illusion — especially when all judges are rooted in the same "cognitive soil." As more and more AI systems rely on models evaluating one another, understanding the independence prerequisites underlying "agreement" matters far more than blindly trusting the vote count.

(Note: This article is based on a discussion thread on Hacker News. The original post contained limited information; the analysis and perspectives here represent the author's extended interpretation drawing on general knowledge in the LLM evaluation field.)