context recall
A metric in the RAGAS evaluation framework that measures how much of the information in the reference answer is covered by the retrieved context.
Timeline (last 90 days)
In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.
In the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.
The author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.
Switching the judge model moved recall by 0.31 and faithfulness by 0.26, three times and nearly twice their respective thresholds, showing that the judge model has a greater impact than the k parameter.
For recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.
The author has open-sourced the full experimental code and raw judge outputs, and plans to rewrite the 13 single-sentence reference answers as multiple claims and re-judge, to verify whether this reduces judge noise on recall.
All Facts (6)
In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.
50%UnverifiedIn the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.
50%UnverifiedThe author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.
50%UnverifiedSwitching the judge model moved recall by 0.31 and faithfulness by 0.26, three times and nearly twice their respective thresholds, showing that the judge model has a greater impact than the k parameter.
50%UnverifiedFor recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.
50%UnverifiedThe author has open-sourced the full experimental code and raw judge outputs, and plans to rewrite the 13 single-sentence reference answers as multiple claims and re-judge, to verify whether this reduces judge noise on recall.
50%