Unverified50% confidenceFactExact time
Systems with the same overall accuracy may produce markedly different error distributions across two population groups, for example frequently wrongly rejecting applicants in group A who would have repaid on time, and more often wrongly approving applicants in group B who ultimately defaulted.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn as many as 28% of system comparisons, ignoring shared reviewer errors leads to the conclusion that 'system A is significantly better than system B', while this significant difference disappears once shared errors are taken into account.79% similarUnverifiedWhen evaluating a binary verdict system, one must report both the acceptance rate of wrong answers and the rejection rate of correct answers (i.e., both denominators); otherwise a single pass rate will mask the failure modes.74% similarUnverifiedThe fairness standard ProPublica used is "group balance in predictive error rates," while the company that developed COMPAS countered that the system is calibrated on "predictive accuracy" for both groups, i.e., the actual recidivism rate corresponding to a high risk score is roughly the same across races.73% similarUnverifiedMemory benchmark scores can fluctuate enormously depending on the evaluation protocol, and the same system may differ by more than ten percentage points under a different evaluation process.73% similarUnverifiedFor an invoice classification model with 97% accuracy, when a reviewer examines 300 invoices per day, only about 9 will be wrong, and errors are buried among correct items, causing the reviewer's rejection rate to approach zero.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1060063API
curl https://kongchang.com/api/v1/knowledge/claims/1060063MCP
get_claim(id=1060063)