Unverified50% confidenceFactExact time
Human calibration (n=46) yields a dialect agreement of κ=1.0 and a power-relevance agreement of κ=0.85, indicating that the automatic evaluation closely matches expert judgment.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
11/4/2026
First Seen
Valid until: 2/2/2027
Sources
Related Entities
Related Claims
UnverifiedShopify calibrated the agreement between LLM judges and human judgment from kappa 0.02 to 0.61, while agreement between humans is 0.69.71% similarUnverifiedThe paper's core evaluation metric is the agreement between LLMs and human annotators.66% similarUnverifiedAcademic research shows that when scoring criteria are strictly restricted to binary pass/fail standards, automated scorers can provide scalable, reproducible scores that are highly correlated with expert human judgment.66% similarUnverifiedAgreement between the deterministic checker and blind annotators was Cohen's kappa = 0.89.65% similarUnverifiedSimilarity measures wording rather than truth; five samples may unanimously agree on a wrong retry count and still score high, so human review of MODEL-DRAFT is still required.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1134118API
curl https://kongchang.com/api/v1/knowledge/claims/1134118MCP
get_claim(id=1134118)