Unverified50% confidenceFactExact time
In LLM judge scenarios, kappa can reveal bias that accuracy masks—for example, when the judge passes everything, accuracy may be high, but kappa will be close to 0, indicating that the judge has no discriminative ability.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe second term (1-M/N)(1-1/K) of the error rate lower bound corresponds to the traditional coverage problem, where the model can only guess randomly for facts that never appeared during training, with a correct-guess probability of 1/K.65% similarUnverifiedIn the LLM-as-a-judge approach, the implicit assumption behind using majority vote or averaging across multiple judges to improve reliability is that the judges are independent of each other and can spot each other's errors.64% similarUnverifiedSPLIT's formula explains a counterintuitive phenomenon: under certain conditions, adding more LLM judges actually reduces accuracy, because verifier errors are correlated, and calling the same or similar verifiers multiple times cannot independently eliminate errors and may instead amplify systematic bias.64% similarUnverifiedAccuracy only checks whether the highest-probability option is correct, while NLL also measures the probability assigned to the correct option, so NLL can worsen while accuracy rises.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1066494API
curl https://kongchang.com/api/v1/knowledge/claims/1066494MCP
get_claim(id=1066494)