The Truth About AI Graders: How Unreliable Is Scoring Without an Answer Key?

Using another model to grade an AI's output has become standard practice for many teams evaluating their systems. But how trustworthy is that score? After reviewing 77 related studies, one author found that without a verified answer key, graders are least reliable precisely on the questions they themselves can't answer correctly — and their errors systematically lean toward being too lenient.

The Grader's Blind Spot: It Can't Judge What It Can't Solve

Before diving into the research, the author wrote down his own expectations: graders should perform well on easy chat-style preference judgments, and fail on hard problems and vertical domains they can't solve themselves. After going through all 77 studies, he found a key fact had been overlooked — the academic community proposed a fix as early as 2023: "give the grader a reference answer." Subsequent research then revealed when this works and when it quietly fails.

The core issue is a repeatedly validated pattern: a grader's ability to judge correctness is highly correlated with its own ability to solve problems. According to JudgeBench research, a grader's capacity to "check answers" is tightly bound to its capacity to "solve problems." When GPT-4o scores, its agreement with experts on questions it answered correctly reaches 0.78 (Cohen's κ); on questions it got wrong, agreement plunges to 0.30. In other words, the less a grader understands, the more likely it is to give unreliable judgments — and the places it doesn't understand are often exactly where accurate evaluation is needed most.

Cohen's κ (Cohen's kappa) is a statistic measuring the agreement between two raters judging the same set of items, typically ranging from 0 to 1 (it can also be negative, indicating below-chance agreement). Its key advantage is that it subtracts the portion of agreement that could occur "purely by luck": if two raters independently assigned scores at the same proportions randomly, raw agreement could be high, but κ removes this chance agreement. Thus κ=0.78 indicates high agreement even after removing chance, while κ=0.30 is only "fair" or even "slight" agreement.

In the grader context, κ reflects true reliability better than raw agreement, because pass/fail label distributions in evaluation tasks are often imbalanced — for example, when the vast majority of answers are correct, a grader that mindlessly judges "pass" will still show high raw agreement. κ exposes this inflation. However, κ is also affected by label distribution: when one class is extremely rare, κ gets depressed or even produces paradoxes. So when reading evaluation research, whenever you see κ, also pay attention to the class proportions in the sample, and ideally combine it with the confusion matrix's false pass and false fail counts rather than fixating on a single number.

Why the Answer Key Is Essential

In 2023, the authors of MT-Bench proposed a simple approach: have the grader answer the question itself first, then use its own answer as the reference. On math problems, this reduced the failure rate from 70% to 15%. That same year, the Prometheus authors also found that with a reference answer, the grader no longer needed to solve the problem itself.

But this method has a fatal precondition: the grader's own answer must be correct. A study spanning finance, math, and reasoning problems, scored by domain experts, showed that without a correct answer for reference, the grader only agreed with experts on questions it could answer correctly itself. The GPT-4o example is telling: without an answer key, agreement was 0.78 on questions it got right and only 0.30 on questions it got wrong; on questions it got wrong, using its own (incorrect) answer as the key actually dropped agreement to 0.16; swapping in the expert's answer raised it to 0.83.

The key phrase is "verified" answer key. Among the five graders the author tested, if the answer key had minor errors, the score was only 0.21 — lower than giving no key at all (0.46), and lower than giving an entirely irrelevant key (0.50). Human-verified model-generated answers scored 0.61, close to human-written answers at 0.69. The author concluded that, in his experimental setup, verified model answers were sufficient.

Errors Lean Toward Leniency, Not False Rejection

Grader errors are not evenly distributed — they have a clear directionality. The author found 19 studies reporting which direction graders err toward: 12 lean toward passing work that should have failed, 3 lean the opposite way, and 4 are mixed.

University-level proof grading illustrates this best. Even when the grader had the expert's grading rubric in hand, 5 of 7 still passed 38.0% to 45.7% of proofs that experts judged as failures, while only falsely rejecting 5.0% to 12.3% of correct proofs. The author pointed out a key distinction: "A rubric tells you what to look for, but that's not the same as telling you what the answer is."

Code review is the opposite. When checking code against written specifications, GPT-4o falsely rejected 26.2% and 35.9% of correct programs on two benchmarks, and if also asked to explain and fix the code, the false rejection rates soared to 73.2% and 87.9%. So if your grader comments on or fixes code, you should also count its false rejections.

The conclusion is clear: false passes and false fails must be counted separately. Looking only at overall agreement masks directionality, and a lenient grader makes your system look better than it actually is.

False pass and false fail are two errors in opposite directions, analogous to false positives and false negatives in statistical testing. A false pass means the grader judges an actually substandard answer as passing; a false fail means it judges an actually passing answer as failing. The two harm the system in completely different ways: false passes let bad answers contaminate training data, shipped results, or leaderboards, inflating perceived system performance, and this contamination is often hard to trace back; false fails discard large numbers of usable positive samples or wrongly penalize correct answers, reducing iteration efficiency and potentially causing teams to misjudge model capabilities pessimistically.

The emphasis on "counting separately" is precisely because overall agreement cancels the two error types against each other. If a grader has many false passes and many false fails, overall accuracy may look merely moderate, but in reality it's unusable in both directions. In practice, you should calculate false pass rate (proportion of actual failures judged as passing) and false fail rate (proportion of actual passes judged as failing) separately, and set different tolerance thresholds based on business cost: in scenarios like content safety and medical advice, the cost of a false pass far exceeds a false fail, so err on the strict side; in scenarios requiring large-scale positive sample mining, focus on suppressing false fails.

Subjective Judgment: The Hard Problem With No Standard Answer

Objective correctness can have an answer key, but subjective judgment cannot — and this is where the author's predictions partially held up.

Performance in vertical domains is mixed. On science questions answered from literature, the best grader, o3, agreed with human voting 65.1% of the time, versus 50% for random guessing. In clinical scoring, the best model reached the level of agreement doctors have among themselves on 3 of 5 task categories, but no single model covered all 5; the author's grader fine-tuned on a small number of doctor-scored cases reached 3 of them. In HealthBench, doctors first wrote grading rubrics, and OpenAI's GPT-4.1 grader exceeded the average doctor's level on 5 of 7 topics.

The author originally expected graders to be solid on easy chat preference judgments, but this prediction didn't fully hold either. GPT-4's agreement with expert annotators was 85%, exceeding the 81% humans have among themselves (after removing ties); but counting ties, the first round was only 66% (pairwise scoring) and 60% (single-answer scoring), while humans were at 63% — roughly human-level. A stricter test found that no model could pass the test of being a "statistically viable replacement for an annotator."

Inter-annotator agreement is the baseline in subjective tasks for measuring "whether humans themselves are reliable." The logic is: if two human experts agree on only 63% of the same samples, a grader achieving 66% agreement with one expert has actually reached the level of "roughly equivalent to another human annotator," rather than being clearly worse. In other words, graders on subjective tasks need not aim for near-100% — humans themselves may not achieve that — and should instead treat achievable inter-annotator agreement as a reasonable target.

But there are two traps here. First, the agreement target depends on the task's decidability: factual Q&A, code review, and other tasks with objective grounding have inherently high human agreement, and graders should approach this high bar; open-ended creative writing, value judgments, and similar tasks have a low agreement ceiling, and demanding high agreement from graders is unfair. Second, high agreement does not equal high correctness. Two annotators may share the same systematic bias, agreeing with each other while both deviating from the true standard. Therefore, in subjective tasks, beyond reporting the grader's agreement with humans, you should also report human-to-human agreement as a control, and ideally use multiple annotators, take majority votes, or compute agreement intervals, rather than aligning to a single "expert's" standard answer.

What Else Sways the Verdict

Beyond the answer key, several hidden factors can destabilize evaluation results.

Order effects: Swapping the order of two answers caused GPT-5.4 to reverse its verdict 11.4% of the time on one benchmark and 10.5% on another; Gemini 3.1 Pro and Claude Opus 4.6 were between 2.0% and 3.8%.

Format effects: With the same content reformatted in Markdown, 4 of 5 graders preferred Markdown over plain text at rates of 73% to 97%, while two human annotators showed only a 57% preference rate on 30 pairs.

Self-favoritism: Whether graders favor their own model's answers remains disputed. Some research suggests GPT-5 is especially lenient toward its own wrong answers, while another study attributes most self-favoritism to the grader's uncertainty on hard problems. If your grader and the model being evaluated are the same, it's best to specifically test that configuration.

The effectiveness of popular fixes is overstated: A panel of three smaller graders beat GPT-4 alone on three QA sets, but only matched its own best member. And when the grader itself isn't stronger than the model it's evaluating, using a small number of human labels to correct it can at best halve the human labels you need.

Order effects and format effects are non-content biases in evaluation — the answer's semantics don't change, only its layout or presentation order, yet the verdict shifts. Order effects are most common in pairwise comparison: the grader is asked to pick the better of answers A and B, and when the two are swapped, if it still prefers whichever one is placed first, that shows it's driven by position rather than quality. Such bias introduces systematic noise into large-scale automated scoring, especially when the model being evaluated always happens to be placed in a certain position — the results will shift overall.

Format effects manifest as the grader's preference for Markdown headings, bullet points, bold text, and other structured formatting. The article's data shows 4 graders preferred Markdown at rates of 73% to 97%, while human annotators were only at 57%, indicating models mistake "looks more polished" for "content is better." The practical takeaway: when comparing different systems, you must standardize format and presentation order; for A/B tests, it's best to randomize left/right positions and alternate formats, otherwise what you're measuring may just be layout differences. A low-cost control method is to add "same answer with changed format/order" probe samples to the evaluation pipeline; if the verdict flip rate is high, it means the grader's judgments on this task aren't stable enough, and conclusions should be treated with caution.

A Grader Check You Can Actually Run

Based on Hamel Husain and Eugene Yan's practical guides, the author expanded a five-step check with 20 samples. It's a screening that can catch a lenient grader, but it can't "clear" a grader — to truly trust one, Husain's guide recommends about 100 samples per failure mode.

1. Pick 20 items your grader is already scoring, at least 10 of which are answers you know to be wrong, then score these 20 items yourself.
2. Run the grader twice: first without an answer key, then using your verified answers as the key. For open-ended questions, give it a checklist of "what a good answer must do."
3. For questions with a unique correct answer, have the grader answer them separately in independent conversations. If it gets more than one or two wrong and passes those wrong answers, don't use it for this type of question without an answer key.
4. Reformat the same answers (headings, bullet points, bold), swap order if scoring pairwise, and count how many verdicts changed.
5. Count false passes and false fails separately. If verified answers eliminate false passes, equip your grader with an answer key; if false passes persist, either fix the answer key or keep human review for this type of item.

Key output metrics include: number of known-wrong items, false passes without a key, false passes with a key, false fails, agreement rates separately on questions the grader got right vs. wrong, and format flip count. In the author's own demonstration run of 24 arithmetic problems, under strict rules he got 4 known errors, 3 false passes without a key, 0 false passes with a key, and 2 format flips, with the entire run costing $2.83.

A Reproducible Conclusion

The value of this analysis isn't in giving a specific number, but in revealing a neglected evaluation blind spot: the score you use to evaluate AI may itself be an unevaluated number. A grader's capability boundary is entangled with its own problem-solving ability, and its errors systematically lean toward "making the system look better."

The author offers an actionable judgment: first equip the grader with a verified answer key, then count what it actually let through. On questions where there's no answer key and the grader can't solve them either, the safest approach is still to keep human review — because it's precisely in these places that automated evaluation is most likely to give a seemingly reasonable but actually wrong sense of reassurance.

One-line Summary

77 studies show: an AI grader's reliability depends on whether it can solve the problems itself, and its errors systematically lean toward leniency.

Paragraph Summary

After reviewing 77 grader studies, the author finds that a grader's ability to judge correctness is tightly bound to its own problem-solving ability: on questions it answers wrong, agreement with experts plunges from 0.78 to 0.30. The answer is that a verified answer key can significantly fix this, but an answer key with minor errors scores only 0.21 — lower than giving no key at all (0.46). Grader errors have directionality: 12 of 19 studies lean toward false passes, and in university proof grading, even with a rubric, 38% to 45.7% of failed proofs were wrongly passed. Order and format effects also significantly change verdicts. Based on this, the author proposes a five-step, 20-sample grader check, and emphasizes that on questions with no answer key that the grader can't solve, human review should still be retained.