JudgeBench
A benchmark for evaluating the judgment accuracy and confidence calibration of AI judge systems. It measures model reliability by computing the actual correctness rate within binned confidence intervals.
Timeline (last 90 days)
According to JudgeBench research, when GPT-4o scores, on questions it itself answers correctly, its agreement with experts reaches 0.78 (Cohen's κ); on questions it answers incorrectly, agreement drops sharply to 0.30.
JudgeBench and ContextualJudgeBench are standardized benchmark test sets used to evaluate the judgment quality of AI judge models.
On JudgeBench, Decider made 620 judgments and publicly disclosed 6 first-round failures.
Decider reports a correct rate of 99.6% for judgments at 90% confidence and above on JudgeBench, 94.0% for the 80–90 interval, 84.1% for the 70–80 interval, and 67.7% for below 70%.
All Facts (4)
According to JudgeBench research, when GPT-4o scores, on questions it itself answers correctly, its agreement with experts reaches 0.78 (Cohen's κ); on questions it answers incorrectly, agreement drops sharply to 0.30.
50%UnverifiedJudgeBench and ContextualJudgeBench are standardized benchmark test sets used to evaluate the judgment quality of AI judge models.
50%UnverifiedOn JudgeBench, Decider made 620 judgments and publicly disclosed 6 first-round failures.
50%UnverifiedDecider reports a correct rate of 99.6% for judgments at 90% confidence and above on JudgeBench, 94.0% for the 80–90 interval, 84.1% for the 70–80 interval, and 67.7% for below 70%.
50%