The Hidden Trap of LLM Evaluator Consensus: How Correlated Errors Inflate Evidence Strength

When Multiple AI Evaluators Agree, Is It Really More Reliable?

Using large language models as automatic evaluators (LLM-as-a-Judge) has become the dominant approach for assessing model outputs and comparing system quality. A widely accepted assumption is that when multiple LLM judges reach consensus, the conclusion becomes more trustworthy. After all, majority voting is a common-sense mechanism humans use to validate decisions.

But a new paper from arXiv (arXiv:2609.22512) throws cold water on this assumption. The core argument cuts right to the heart of the matter: consensus is only strong evidence when the errors made by individual judges are independent of one another. In practice, LLM judges are often trained on similar data and evaluate in similar ways — meaning they tend to make the same mistakes. When errors are correlated, consensus is no longer as strong a signal as it appears.

The rise of the LLM-as-a-Judge paradigm has a practical rationale: human evaluation is expensive, slow, and hard to scale, while well-trained LLMs can achieve near-human judgment quality on many evaluation tasks. The typical approach involves using models like GPT-4 or Claude as judges — scoring candidate responses or performing pairwise comparisons — with majority voting across multiple judges used to strengthen the robustness of conclusions. This paradigm is widely used in leaderboards like LMSYS Chatbot Arena, in RLHF reward model training, and in automated evaluation pipelines across countless research papers. The statistical basis for trusting majority voting is Condorcet's Jury Theorem: if each voter independently makes the correct judgment with probability greater than 50%, then the more voters there are, the more the collective decision converges toward the correct answer. But the theorem's critical assumption — that voters are mutually independent — is precisely what this research calls into question.

Ten Judges Are Worth Only 3.5 Independent Ones

The research team quantified this error correlation, and the results are striking. Within their primary pool of ten judges, the average pairwise error correlation coefficient was 0.21. This means the judges are not making independent assessments — when one judge makes an error, others are more likely to make the same error in the same place.

The implication becomes even more impactful when converted into statistical information: the ten judges together provide roughly the same statistical information as 3.5 truly independent judges. In other words, you may think you've stacked ten evaluators to boost reliability, but your effective sample size has shrunk by nearly two-thirds.

What's especially concerning is that this dependency is actually stronger among high-accuracy frontier judges — even when those judges come from different providers. The intuition that cross-validating with top models from different companies and architectures avoids shared-source bias turns out to be overstated. As frontier models converge in capability, their error patterns converge as well.

Understanding "effective number of independent judges" requires a bit of statistical intuition. When N judges are completely independent, the information they provide scales proportionally with N. But when errors are correlated, the actual information shrinks according to the degree of correlation — this can be measured using effective sample size. A useful analogy: if you survey 100 people about their political views, but all 100 come from the same neighborhood and are exposed to the same media, their collective opinion may be equivalent to only a handful of truly independent observers. A correlation of 0.21 may seem modest, but in multi-judge aggregation settings, even small positive correlations quickly deflate effective sample size — which is the mathematical reason behind the "ten becomes three-point-five" finding. The researchers attribute the stronger correlation among frontier models to the heavy overlap in training data (large-scale crawls of the same internet corpora) and the shared fitting of human preference data through RLHF, which causes top models to homogenize in both "aesthetic preferences" and error patterns.

Ignoring Shared Errors Can Lead to Wrong Conclusions

This error correlation doesn't just cause theoretical information loss — it can directly distort real-world comparison conclusions. The study found that in as many as 28% of system comparisons, ignoring shared errors leads to a conclusion that "System A is significantly better than System B"; once shared errors are factored in, that apparent significant difference disappears.

This is a wake-up call for the entire model evaluation ecosystem. Many papers, leaderboards, and A/B tests rely on majority voting from LLM judges to claim that one model is "better." If nearly a third of significant findings cannot survive correction for error correlation, then a large number of conclusions based on evaluator consensus may represent overstated evidence — which is precisely the core problem the paper's title points to.

How Errors Are Distributed Matters Too

The research also reveals a dimension that's easy to overlook: measuring the total amount of correlation isn't enough — the distribution pattern of errors matters just as much.

The paper distinguishes between two types of shared errors: those committed by the majority of judges, and those concentrated among a small subset of judges. These two types affect consensus differently, and each is better suited to different voting methods. In other words, even with the same overall correlation coefficient, whether errors "spread widely" or "cluster locally" points toward different optimal aggregation strategies. This means a single correlation metric cannot provide a one-size-fits-all correction for consensus reliability.

The two error patterns the paper distinguishes can be compared to "common cause" errors in epidemiology. Widely shared errors are analogous to systematic bias: most judges make the same directional mistake on certain types of questions — such as preferences around code style or overvaluing long answers. These errors are difficult to eliminate through aggregation because they won't be corrected by a minority. Locally clustered errors, by contrast, resemble subgroup bias: only models from a specific provider or architecture make certain types of mistakes. In this case, deliberately selecting judges from different subgroups can allow majority voting to cancel errors out. Identifying which distribution pattern applies to your judge pool is the key prerequisite for choosing weighting strategies, weighted schemes, or stratified sampling — and it's the core logic behind the paper's recommendation to "diagnose first with a trusted sample, then choose a method."

A Practical, Actionable Solution

To its credit, this research doesn't stop at problem identification — it also proposes a simple, actionable methodology:

• Estimate judge accuracy using a small trusted sample: Large-scale annotation isn't necessary; a batch of high-quality, reliable examples is sufficient.
• Identify shared errors: Use these trusted examples to find where judges tend to make the same mistakes.
• Account for shared errors when analyzing results: Rather than naively treating each judge as an independent information source when tallying votes.
• Choose a voting method in advance: Before applying an aggregation method to new data, use the trusted sample to determine which voting approach is most appropriate.

The value of this workflow lies in its manageable cost. It doesn't require abandoning the LLM-as-a-Judge paradigm — it simply adds a necessary statistical correction, restoring evaluator consensus to its appropriate level of evidential strength.

Implications for Practitioners

For researchers and engineering teams that rely on automated evaluation, this study delivers at least three practical signals. First, don't naively equate the number of judges with strength of evidence — the marginal returns of adding more judges may be far lower than expected. Second, cross-validating with top models from different providers does not eliminate error correlation — it may actually be worse. Third, before claiming "significantly better," apply corrections for error correlation — otherwise the reliability of your conclusions is questionable.

At a time when LLM judges have become fundamental evaluation infrastructure, this research reminds us: consensus has value, but the "quality" of that consensus needs to be carefully measured rather than taken for granted.