context precision
A metric in the RAGAS evaluation framework that checks how many of the retrieved chunks are genuinely relevant and at what ranking positions the relevant chunks appear.
Timeline (last 90 days)
In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.
When only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.
In the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.
The author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.
For recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.
Of about 1,200 judge calls, two failed due to JSON escaping format errors; on one day the judge's responses became ten times slower and context precision timed out on 16 of 18 questions, because RAGAS's timeout limit covers all calls for a single question and single metric, and precision makes 8 judge calls per question when k=8.
All Facts (6)
In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.
50%UnverifiedWhen only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.
50%UnverifiedIn the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.
50%UnverifiedThe author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.
50%UnverifiedFor recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.
50%UnverifiedOf about 1,200 judge calls, two failed due to JSON escaping format errors; on one day the judge's responses became ten times slower and context precision timed out on 16 of 18 questions, because RAGAS's timeout limit covers all calls for a single question and single metric, and precision makes 8 judge calls per question when k=8.
50%