Unverified50% confidenceBenchmarkExact time
On two corpora each containing 900 responses and a total of eight scorers, the compliance rate differs by 22 to 25 times across scorers.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAfter excluding the three collapsed scorers, the compliance rates on the four remaining corpora still differ by 1.7 to 2.8 times, and the non-refusal rates differ by 10 to 22 percentage points.73% similarUnverifiedCross-grader differences in compliance rates manifest as multiplicative (ratios can reach tens of times), whereas differences in non-refusal rates manifest as additive (percentage-point differences), so non-refusal rate is more robust to grader disagreement but coarser-grained.73% similarUnverifiedIn the configuration using ten policy snapshots, using only 8 prompts achieved an average accuracy of 24.09%, approaching the 24.51% achieved with 14,080 prompts.73% similarUnverifiedOne scorer in the study never once assigned the compliant label across 900 responses.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074739API
curl https://kongchang.com/api/v1/knowledge/claims/1074739MCP
get_claim(id=1074739)