Why Do LLM Judges So Often Disagree With Experts? How a Mixture-of-Judges Framework Breaks the Deadlock

When LLM Judges Start "Going Rogue"

In the era of large language models (LLMs), one increasingly popular practice is using LLMs as "judges" (LLM-as-a-Judge) to automatically evaluate the output quality of other models or dialogue systems. The approach seems efficient — after all, having machines evaluate machines is cheap, fast, and scalable. Yet a new study reveals a sobering truth: your LLM judge may be seriously out of step with human expert judgment.

This work introduces a "reference-full" evaluation benchmark that is remarkably rare in both scale and authenticity within the dialogue evaluation space. It brings together hundreds of complete human-to-human conversations written by professional screenwriters, with realistic conversational turn density, and features over 36,000 expert-generated dialogue turns annotated with more than 36,000 turn-level human labels.

In other words, this is not yet another benchmark papering over its shortcomings with synthetic data — it is a rigorous dataset grounded in real human conversations, scored turn by turn by domain experts.

The "Inherent Flaws" of Existing Evaluation Frameworks

Built for the Wrong Task

The study highlights a critical issue: most mainstream dialogue evaluation frameworks are, in fact, not designed for dialogue. They were originally built to handle tasks like summarization, translation, and short-form QA.

These tasks differ fundamentally from multi-turn dialogue — they typically involve single-turn input-output pairs, well-defined objectives, and relatively closed-ended answers. Real conversation, by contrast, is dynamic, context-dependent, and rich with pragmatic nuance. Applying metrics designed for summarization directly to dialogue evaluation is like using a ruler to measure weight.

"Validating Themselves" on Synthetic Data

The deeper problem lies in how these metrics are validated. The study emphasizes that many evaluation metrics are derived and validated on synthetic data rather than real human conversations.

This creates a self-reinforcing loop: a metric performs well on artificial data, is deemed "effective," but when confronted with complex real-world conversations judged by human experts, that "effectiveness" can collapse instantly. Synthetic data lacks the noise, ambiguity, emotion, and contextual discontinuity found in genuine dialogue, causing metrics to severely overfit to idealized scenarios.

"Failures" Under Large-Scale Testing

When researchers put these methods to the test against an unprecedented scale of expert judgments, the results were discouraging:

Both classic automated metrics and reference-free LLM-as-a-Judge approaches were found to be unreliable.

Two points deserve particular attention:

• Classic automated metrics fail: Traditional n-gram overlap metrics like BLEU and ROUGE have always struggled with open-ended dialogue, and this study further exposes their weaknesses when measured against large-scale human annotations.
• Reference-free LLM judges are equally unreliable: "Reference-free" means letting an LLM score outputs based on its own understanding without being given ground-truth answers. This is widely regarded as the most flexible and modern evaluation approach — but the study shows its correlation with expert judgment is equally questionable.

This finding carries an industry-wide warning: many teams are using unreliable LLM judges to make product decisions, model selection choices, and deployment evaluations — potentially building on fundamentally flawed signals.

The Solution: A Mixture-of-Judges Framework

Faced with the unreliability of any single evaluation signal, the study proposes a more robust solution — the Mixture-of-Judges framework.

The core idea is straightforward: rather than relying on any single source of evaluation, fuse multiple assessment signals together. Different evaluation dimensions, different model perspectives, and different metrics each have their own blind spots — but through principled combination, they can complement one another and reduce bias.

The results are quite striking: compared to any single method, this framework improves correlation with human evaluation by approximately 30%.

The insight behind this number is that evaluation itself needs an ensemble learning mindset. Just as ensemble models typically outperform individual ones, evaluation systems should be a combination of multiple signals rather than a reckless bet on a single "omniscient judge."

Practical Takeaways for Practitioners

For teams building or using LLM-based evaluation systems, this research offers several pragmatic recommendations:

1. Be wary of metrics validated only on synthetic data. If an evaluation method has only proven itself on synthetic data, its performance on real conversations deserves skepticism.
2. Don't blindly trust a single LLM judge. Reference-free LLM scoring is convenient, but it may systematically diverge from expert judgment.
3. Embrace multi-signal fusion. Combining multiple evaluation signals to improve alignment with human judgment is currently the more reliable path forward.
4. Value the worth of real data. High-quality data generated by professionals and annotated turn by turn remains the irreplaceable gold standard for calibrating evaluation systems.

Conclusion

LLM judges are not a universal solution. As we grow increasingly reliant on "having AI evaluate AI," we need to remain clear-eyed about the limitations of these judges themselves. Through expert annotations at an unprecedented scale, this research punctures some of the illusions surrounding automated evaluation — while also pointing toward a more robust path forward: using a Mixture-of-Judges framework to better approximate genuine human judgment. In the pursuit of automated evaluation, alignment with expert opinion should always remain the non-negotiable benchmark.