Unverified50% confidenceFactExact time
Among 19 studies that reported which direction the evaluator's errors lean, 12 lean toward passing work that should have failed, 3 lean in the opposite direction, and 4 are mixed.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAmong the panel of ten LLM reviewers used as the primary reviewer pool in the study, the average pairwise correlation coefficient of reviewer errors was 0.21.76% similarUnverifiedResearch shows that in certain tasks, simply changing '请判断' to '请分析' in a prompt can cause accuracy fluctuations of 5–10 percentage points.73% similarUnverifiedIn university-level proof grading, even with the experts' grading rubric in hand, 5 of 7 evaluators still passed 38.0% to 45.7% of proofs that experts judged as failures, while falsely failing only 5.0% to 12.3% of correct proofs.72% similarUnverifiedThe accuracy data in the test has an error margin of approximately ±8–10 percentage points at n=100 sample size (based on a 95% confidence-level binomial confidence interval).71% similarUnverifiedIn 11 of the 12 models, between 87.5% and 100% of capitulation cases stopped at the exact wrong number provided by the user, meaning capitulation typically involves switching sides entirely.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1026030API
curl https://kongchang.com/api/v1/knowledge/claims/1026030MCP
get_claim(id=1026030)