Unverified50% confidenceBenchmarkExact time
In the example, for the same set of human labels (18 pass, 2 fail), a lazy judge that passes everything gets an accuracy of 0.90 but a kappa of 0.00; a better judge gets an accuracy of 0.95 but a kappa of only 0.77, still not reaching the 0.80 threshold.
1
Sources
50%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn the Phase 1 post-hoc comparison, SAGE's evaluation results were compared against 105 human reviews from the original competition, yielding an ordinal consistency of kappa = 0.29, which falls in the "fair" range.75% similarUnverifiedFor an invoice classification model with 97% accuracy, when a reviewer examines 300 invoices per day, only about 9 will be wrong, and errors are buried among correct items, causing the reviewer's rejection rate to approach zero.74% similarUnverifiedThe accuracy data in the test has an error margin of approximately ±8–10 percentage points at n=100 sample size (based on a 95% confidence-level binomial confidence interval).74% similarUnverifiedAmong the five evaluators the authors tested, if the answer key had minor errors, the score was only 0.21 — lower than giving no key at all (0.46) and lower than giving a completely irrelevant key (0.50); while a human-verified model-generated answer scored 0.61, close to the 0.69 of a human-written one.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1035667API
curl https://kongchang.com/api/v1/knowledge/claims/1035667MCP
get_claim(id=1035667)