Unverified50% confidenceBenchmarkExact time
When only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/19/2026
First Seen
Valid until: 1/17/2027
Sources
Related Entities
Related Claims
UnverifiedThe author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.78% similarUnverifiedIn the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.77% similarUnverifiedEven with the judge's temperature set to 0, judging the same answers and chunks three times, the 18-question mean of precision fluctuated by 0.085, faithfulness by 0.042, recall by 0.046, and only relevancy stayed relatively stable at 0.003.76% similarUnverifiedIn RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1065189API
curl https://kongchang.com/api/v1/knowledge/claims/1065189MCP
get_claim(id=1065189)