Unverified50% confidenceFactExact time
When only 5% of outputs fail, an evaluator that passes everything agrees with the expert 95% of the time, yet its recall is zero, which is called the "agreement paradox."
1
Sources
50%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor an invoice classification model with 97% accuracy, when a reviewer examines 300 invoices per day, only about 9 will be wrong, and errors are buried among correct items, causing the reviewer's rejection rate to approach zero.79% similarUnverifiedIn the example, for the same set of human labels (18 pass, 2 fail), a lazy judge that passes everything gets an accuracy of 0.90 but a kappa of 0.00; a better judge gets an accuracy of 0.95 but a kappa of only 0.77, still not reaching the 0.80 threshold.77% similarUnverifiedA 95% confidence level means that if the same methodology were used to conduct 100 surveys, approximately 95 of them would produce results falling within the margin of error.77% similarUnverifiedThe most consistent evaluator's agreement rate with the true answers is only 51%, meaning it is "consistently wrong."77% similarUnverifiedThe accuracy data in the test has an error margin of approximately ±8–10 percentage points at n=100 sample size (based on a 95% confidence-level binomial confidence interval).75% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1036262API
curl https://kongchang.com/api/v1/knowledge/claims/1036262MCP
get_claim(id=1036262)