Unverified50% confidenceFactExact time
Switching the judge model moved recall by 0.31 and faithfulness by 0.26, three times and nearly twice their respective thresholds, showing that the judge model has a greater impact than the k parameter.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/19/2026
First Seen
Valid until: 1/17/2027
Sources
Related Entities
Related Claims
UnverifiedIn the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.69% similarUnverifiedAfter the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.67% similarUnverifiedEven with the judge's temperature set to 0, judging the same answers and chunks three times, the 18-question mean of precision fluctuated by 0.085, faithfulness by 0.042, recall by 0.046, and only relevancy stayed relatively stable at 0.003.64% similarUnverifiedIn LLM-as-a-judge evaluation, the judge's judgment is highly dependent on the wording of the prompt, and different descriptions of evaluation criteria can cause the pass rate to plummet from 100% to 0%63% similarUnverifiedRewriting one criterion in a prompt causes the pass rate of other criteria within the same prompt to fluctuate by 5 to 10 percentage points.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1065194API
curl https://kongchang.com/api/v1/knowledge/claims/1065194MCP
get_claim(id=1065194)