Unverified50% confidenceFactExact time
In a preregistered study, confidence intervals relying solely on LLM judges failed to cover the full human-annotation ground truth in 18 of 21 task-and-judge combinations.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe criteria are divided into four families—foundational, premature disclosure, goal deviation, and information omission—with 19 failure modes in total, of which 18 were triggered in actual testing.80% similarUnverifiedIn the benchmark's 60 runs, the early termination conditions were triggered 27 times, including 21 detections of lack of progress and 6 cases of model provider unavailability.78% similarUnverifiedThe developer cautions that the sample size of 18 matchups is insufficient to rank the models reliably.77% similarUnverifiedTheoretically reliable clustered standard error estimation typically requires at least 30–50 clusters, whereas this study has only 19 independent time units, making the sample size relatively small.73% similarUnverifiedThis test has limitations: only 14 groups, one 2-billion-parameter model, one run; three valid paraphrases were corrected before the run because they changed the legal meaning, and the corrections are in the same commit as the test set; the test set has not yet been run on larger models.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1069898API
curl https://kongchang.com/api/v1/knowledge/claims/1069898MCP
get_claim(id=1069898)