Cohen's κ
A statistic measuring the agreement between two raters classifying the same set of samples, computed with the probability of chance agreement removed; it ranges from -1 to 1 and is used to assess whether automated review can replace human review.
Core Facts
Timeline (last 90 days)
A commonly used empirical threshold in research fields: κ above 0.8 is regarded as high agreement, and 0.6 to 0.8 as substantial agreement.
On the training samples, the character classification model achieved a macro-averaged AUC of 0.97 and a Cohen's κ of 0.81.
Cohen's κ is a statistic measuring the agreement between two raters classifying the same set of samples, calculated with the probability of chance agreement subtracted out.
According to JudgeBench research, when GPT-4o scores, on questions it itself answers correctly, its agreement with experts reaches 0.78 (Cohen's κ); on questions it answers incorrectly, agreement drops sharply to 0.30.
All Facts (4)
Cohen's κ is a statistic measuring the agreement between two raters classifying the same set of samples, calculated with the probability of chance agreement subtracted out.
75%UnverifiedA commonly used empirical threshold in research fields: κ above 0.8 is regarded as high agreement, and 0.6 to 0.8 as substantial agreement.
50%UnverifiedOn the training samples, the character classification model achieved a macro-averaged AUC of 0.97 and a Cohen's κ of 0.81.
50%UnverifiedAccording to JudgeBench research, when GPT-4o scores, on questions it itself answers correctly, its agreement with experts reaches 0.78 (Cohen's κ); on questions it answers incorrectly, agreement drops sharply to 0.30.
50%