answer relevancy
A metric in the RAGAS evaluation framework that assesses how relevant a generated answer is to the question, implemented by having an LLM reconstruct the question from the answer and computing embedding similarity.
Timeline (last 90 days)
In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.
When only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.
The author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.
For recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.
For a question asking which cognitive biases the papers involve, one generation honestly said the papers did not name specific biases and relevancy gave it 0; another generation confidently listed a bias that does not exist in the corpus and relevancy instead gave it 0.58, with only faithfulness catching the fabrication (0.17).
All Facts (5)
In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.
50%UnverifiedWhen only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.
50%UnverifiedThe author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.
50%UnverifiedFor recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.
50%UnverifiedFor a question asking which cognitive biases the papers involve, one generation honestly said the papers did not name specific biases and relevancy gave it 0; another generation confidently listed a bias that does not exist in the corpus and relevancy instead gave it 0.58, with only faithfulness catching the fabrication (0.17).
50%