Unverified50% confidenceFactExact time
Even when each scorer is called only once, the results of scorer disagreement remain unchanged; collapse is not statistical fluctuation from multiple sampling.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedCross-grader differences in compliance rates manifest as multiplicative (ratios can reach tens of times), whereas differences in non-refusal rates manifest as additive (percentage-point differences), so non-refusal rate is more robust to grader disagreement but coarser-grained.74% similarUnverifiedAfter excluding the three collapsed scorers, the compliance rates on the four remaining corpora still differ by 1.7 to 2.8 times, and the non-refusal rates differ by 10 to 22 percentage points.73% similarUnverifiedThere is no stable mapping between a model's self-evaluation score and its actual accuracy; rephrase the same prompt and the score may drift substantially.73% similarVerifiedIn scenarios with extreme class imbalance (such as fraud detection), looking solely at accuracy as an evaluation metric is completely meaningless.70% similarUnverifiedThe comparative experiment did not disclose specific dataset sizes, evaluation metrics, or details of comparison fairness, all of which affect the generalizability of its conclusions.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074741API
curl https://kongchang.com/api/v1/knowledge/claims/1074741MCP
get_claim(id=1074741)