Unverified50% confidenceFactExact time
On a 50-sample evaluation set, an accuracy change from 74% to 76% corresponds to just one more correct sample, a difference that is statistically almost meaningless.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe most consistent evaluator's agreement rate with the true answers is only 51%, meaning it is "consistently wrong."79% similarUnverifiedThe article points out that the 50-sample evaluation set is too small, since gaining or losing a single sample can move accuracy by 2 percentage points, so the 86% result may simply reflect the model fitting these 50 samples rather than genuine generalization.79% similarUnverified研究表明人类评估者在判断LLM输出的事实准确性时，正确率往往只有50%-70%78% similarUnverifiedA gating accuracy of 58% reported in the paper was mistakenly treated as end-to-end quality, whereas the true end-to-end quality is 98%.78% similarUnverifiedThe study found that the faithfulness score for counterfactual inputs was only 0.05 points lower than for factual inputs (on a 1–5 scale), a difference that is almost negligible.77% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1062619API
curl https://kongchang.com/api/v1/knowledge/claims/1062619MCP
get_claim(id=1062619)