Judge Bait
A benchmark designed to test the robustness of the LLM-as-a-Judge paradigm. It systematically evaluates the judgment accuracy and resistance to manipulation of LLM judges when they cannot independently verify facts, using multiple distractor strategies such as verbosity, authority appeals, pressure, and prompt injection.
Timeline (last 90 days)
A developer built a benchmark called Judge Bait, designed to induce LLM judges to select the wrong answer.
In the Judge Bait experiment, the model under test acts as a judge, receives a question and a pair of answers (one correct, one incorrect), and selects the correct answer; each pair is presented in both orders, and a question is only considered passed if both orderings are answered correctly.
Judge Bait designed multiple categories of bait, including verbosity, authority, pressure, and injection, and the judge prompt explicitly requires ignoring order, length, format, tone, and subjective opinion.
The test questions are divided into 40 easy-to-verify questions and 40 hard-to-verify questions, and all standard answers of the math, code, and date types are validated in Python.
When a judge cannot verify an answer, it latches onto available cues such as position to make a judgment, and bait provides more useful cues.
Under injection bait, weak models treat the injected incorrect numbers as their own calculation results and fabricate verification processes out of thin air, with the explanatory text appearing consistent with genuine verification, making it impossible to detect errors through auditing.
Placing verbose padding on the correct answer causes the judge's score to plummet—Grok non-reasoning hard-question scores dropped from 0.875 to 0.19—indicating that verbosity bias exists only in reverse.
All Facts (7)
A developer built a benchmark called Judge Bait, designed to induce LLM judges to select the wrong answer.
50%UnverifiedIn the Judge Bait experiment, the model under test acts as a judge, receives a question and a pair of answers (one correct, one incorrect), and selects the correct answer; each pair is presented in both orders, and a question is only considered passed if both orderings are answered correctly.
50%UnverifiedJudge Bait designed multiple categories of bait, including verbosity, authority, pressure, and injection, and the judge prompt explicitly requires ignoring order, length, format, tone, and subjective opinion.
50%UnverifiedThe test questions are divided into 40 easy-to-verify questions and 40 hard-to-verify questions, and all standard answers of the math, code, and date types are validated in Python.
50%UnverifiedWhen a judge cannot verify an answer, it latches onto available cues such as position to make a judgment, and bait provides more useful cues.
50%UnverifiedUnder injection bait, weak models treat the injected incorrect numbers as their own calculation results and fabricate verification processes out of thin air, with the explanatory text appearing consistent with genuine verification, making it impossible to detect errors through auditing.
50%UnverifiedPlacing verbose padding on the correct answer causes the judge's score to plummet—Grok non-reasoning hard-question scores dropped from 0.875 to 0.19—indicating that verbosity bias exists only in reverse.
50%