ContextualJudgeBench
A benchmark for evaluating the judgment consistency of AI judge systems in context-aware scenarios. It uses a pairwise bidirectional protocol requiring the model to judge correctly under both presentation orders to receive credit; the random baseline is 25%.
Timeline (last 90 days)
JudgeBench and ContextualJudgeBench are standardized benchmark test sets used to evaluate the judgment quality of AI judge models.
ContextualJudgeBench tests whether judgments are affected by position bias by presenting items in both orders, counting a case as passed only when both orders are judged correctly, which reduces the random baseline from 50% to 25%.
Decider's self-tested consistent accuracy on ContextualJudgeBench under the official paired protocol is 67.1%, compared to the official reference value of 65.4%, with 12 orders excluded due to repeated platform failures.
All Facts (3)
JudgeBench and ContextualJudgeBench are standardized benchmark test sets used to evaluate the judgment quality of AI judge models.
50%UnverifiedContextualJudgeBench tests whether judgments are affected by position bias by presenting items in both orders, counting a case as passed only when both orders are judged correctly, which reduces the random baseline from 50% to 25%.
50%UnverifiedDecider's self-tested consistent accuracy on ContextualJudgeBench under the official paired protocol is 67.1%, compared to the official reference value of 65.4%, with 12 orders excluded due to repeated platform failures.
50%