Is Your LLM Judge Biased? Three Calibration Steps to Make Evaluations Trustworthy

Title

Is Your LLM Judge Biased? Three Calibration Steps to Make Evaluations Trustworthy

Meta Description

LLM-as-a-Judge suffers from three predictable biases: position bias, verbosity bias, and self-preference. This article presents three fixes: swap-order double judging, rubric-based pass/fail grading, and validating the judge with human labels and confusion matrices—plus reusable Python code.

Article Content

You use LLM-as-a-Judge to score an AI feature, the dashboard shows a 12% quality improvement, and you ship with confidence. Then users report the new version is worse. The problem is often not your application model but the judge itself—it's measuring the wrong thing. LLM judges are useful but have predictable biases; without designing evaluations to counter them, your eval system will quietly reward the wrong answers.

The core principle is just one: treat the LLM judge as another model in production—test it before you trust it.

Three of the Most Common Judge Biases

The MT-Bench / Chatbot Arena paper (Zheng et al., 2023) found that strong judges like GPT-4 can exceed 80% agreement with human preferences, roughly on par with agreement among humans themselves. The same paper also documented failure modes:

1. Position bias: in pairwise comparisons, the judge tends to favor the answer in a certain position, usually the first. Wang et al. showed that merely swapping the order of candidate answers can flip the ranking.
2. Verbosity bias: longer, more detailed-looking answers receive higher scores even when they are not more correct.
3. Self-preference: models tend to score their own outputs (or stylistically similar outputs) higher. Panickssery et al. found that LLM evaluators can recognize their own generations and favor them.

These findings don't mean "don't use LLM judges"—they mean the default setup—"here are two answers, which is better?"—is not yet a valid evaluation.

MT-Bench (Multi-Turn Benchmark) is a multi-turn dialogue evaluation benchmark proposed by Zheng et al. in 2023, containing 80 multi-turn questions and 160 single-turn questions across eight categories including writing, role-play, reasoning, and coding. Chatbot Arena is a crowdsourced battle platform where users vote on responses from two anonymous models, accumulating over a million human-preference data points. The paper used this data to validate GPT-4's agreement with human preferences as a judge and systematically identified failure modes such as position bias, verbosity bias, and self-preference, laying the groundwork for subsequent LLM-as-a-Judge research.

Fix One: Swap the Order, Judge Twice

For pairwise comparisons, swap the positions of the candidate answers and run the judge twice. Only count a win when the conclusion is consistent across both orders; if they disagree, treat it as a tie or escalate to a human.

from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = """You are grading two answers to the same question.
Question: {question}
Answer 1: {a}
Answer 2: {b}
Which answer is more correct and more helpful? Ignore length.
Reply with exactly one token: 1, 2, or TIE."""

def judge_once(question, a, b, model="gpt-4o-mini"):
    r = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(question=question, a=a, b=b)}],
    )
    return r.choices[0].message.content.strip()

def judge_pair(question, old, new):
    first = judge_once(question, old, new)   # old in slot 1
    second = judge_once(question, new, old)  # new in slot 1
    if first == "2" and second == "1":
        return "new"
    if first == "1" and second == "2":
        return "old"
    return "tie"  # the judge changed its mind when order changed

Additionally track the proportion of "ties caused by disagreement." If this proportion is high, the judge is mostly reacting to position, and the scores it previously gave have limited value.

Fix Two: Grade Pass/Fail Against a Written Rubric, Reasoning First

A score from 1 to 10 looks precise, but judges (and humans) are inconsistent about the boundary between 6 and 7. Hamel Husain's LLM judge guide recommends switching to binary pass/fail judgments with written comments, with the rubric co-defined by domain experts.

The rubric itself also counters verbosity bias, because the judge checks specific items rather than an overall impression:

RUBRIC = """Grade the answer to the customer's question.
PASS only if ALL are true:
- It states the correct refund window from the policy below.
- It does not promise anything the policy does not allow.
- It answers in under 120 words.
Policy: {policy}
Question: {question}
Answer: {answer}

First write one sentence of reasoning. Then on a new line write PASS or FAIL."""

Requiring the judge to write a brief reason before reaching a conclusion also gives you readable clues when debugging the judge.

Fix Three: Measure the Judge with Human Labels

This is the step most teams skip. Before letting the judge decide releases:

1. Take 50 to 100 real samples.
2. Have someone with domain knowledge label each sample PASS or FAIL.
3. Run the judge on the same set of samples.
4. Compare the results.

from sklearn.metrics import cohen_kappa_score, confusion_matrix

human = ["PASS", "FAIL", "PASS", ...]   # expert labels
judge = ["PASS", "PASS", "PASS", ...]   # judge verdicts

print(confusion_matrix(human, judge, labels=["PASS", "FAIL"]))
print("kappa:", cohen_kappa_score(human, judge))

Look at the confusion matrix, not just a single number. If 90% of samples are good answers to begin with, a judge that passes everything can still show "90% accuracy." What really matters is whether it catches failure samples. Read through each disagreement, tighten the rubric, rerun, until it is trustworthy on the critical cases.

Eugene Yan's survey of LLM evaluator research is worth reading: judges perform well on some tasks and poorly on others, and the only way to know which your task is, is to measure it.

Cohen's kappa is a statistic measuring agreement between two annotators that corrects for the probability of chance agreement. It is computed as (\kappa = (p_o - p_e) / (1 - p_e)), where (p_o) is the observed agreement and (p_e) is the expected chance agreement. Kappa values are typically interpreted as: below 0 worse than chance, 0-0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial agreement, 0.81-1.00 almost perfect agreement. In the LLM judge setting, kappa reveals biases that accuracy conceals—for example, a judge that passes everything may have high accuracy, but kappa will be near 0, signaling that the judge has no discriminative ability.

A Few Extra Good Habits

• Use a different model family for the judge, ideally not the same lineage as the model generating the answers, to reduce self-preference.
• Pin the judge's model version and temperature. A judge that silently changes is a moving ruler.
• Re-check agreement every time you change the judge prompt or judge model, just as you rerun tests after refactoring code.
• Keep a few known-bad samples in every evaluation. If the judge lets them pass, that evaluation run is invalid.

Key Takeaways

An LLM judge is a model that predicts quality. Give it a clear rubric, eliminate the tractable biases (order, length, self-preference), and measure its performance against human labels before it decides what ships.

Finally, one question for your team: are you currently using pairwise comparisons, 1-to-10 scoring, or pass/fail judgments? Recalibrating with a different approach may reveal real quality changes that were previously masked.

References

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
• Wang et al., Large Language Models are not Fair Evaluators (2023)
• Panickssery et al., LLM Evaluators Recognize and Favor Their Own Generations (2024)
• Hamel Husain, Using LLM-as-a-Judge For Evaluation: A Complete Guide
• Eugene Yan, Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
• scikit-learn, cohen_kappa_score

One-line Summary

LLM judges have position, verbosity, and self-preference biases—fix them by swapping order, using binary rubrics, and calibrating against human labels.

Paragraph Summary

The article points out that LLM-as-a-Judge in AI feature evaluations can suffer from position bias, verbosity bias, and self-preference, distorting evaluation results. It proposes three fixes: for pairwise comparisons, swap the answer order and run twice, counting a win only when conclusions agree; switch to binary pass/fail judgments based on a written rubric and require the judge to write reasoning first; validate the judge with 50-100 human-labeled samples, examining the confusion matrix and Cohen's kappa. It also recommends using a judge from a different model family, pinning version and temperature, re-checking agreement after each change, and retaining known-bad samples. The core principle is to treat the judge as a production model—test before you trust.