Faithfulness
A metric in natural language generation evaluation that measures whether a response is derived strictly from the given context rather than introducing external information. It differs subtly from truthfulness and serves as a more measurable and controllable evaluation standard in RAG scenarios.
Timeline (last 90 days)
Evaluating a RAG system requires measuring two stages separately: context relevance checks the retrieval step (whether the retrieved documents truly contain the answer), and faithfulness checks the generation step (whether the generated answer comes strictly from the retrieved context rather than introducing external information).
Faithfulness corresponds to the concept of factual consistency (Faithfulness / Factual Consistency) in natural language generation evaluation, and differs subtly from truthfulness: truthfulness means whether the answer matches facts in the objective world, while faithfulness only requires that the answer does not go beyond the given context; therefore faithfulness is a more measurable and more controllable metric, whereas truthfulness also depends on the reliability of the retrieval source itself.
Faithfulness checking is done claim by claim, and this claim-by-claim comparison is more precise than overall scoring, because an answer may have 90% of its content verifiable but mixed with 10% fabricated information, and that 10% hallucination is enough to mislead users into making wrong decisions.
Recall@k and MRR are commonly used to evaluate retrieval quality, while evaluating generation typically looks at the consistency between answers and sources, i.e., faithfulness.
The study found that the faithfulness of deletion-based explanations is highly coupled with the detection strength at the time of explanation generation.
Deletion testing evaluates explanation faithfulness by masking the regions highlighted by the explanation and observing the degree of drop in model confidence.
All Facts (6)
Evaluating a RAG system requires measuring two stages separately: context relevance checks the retrieval step (whether the retrieved documents truly contain the answer), and faithfulness checks the generation step (whether the generated answer comes strictly from the retrieved context rather than introducing external information).
50%UnverifiedFaithfulness corresponds to the concept of factual consistency (Faithfulness / Factual Consistency) in natural language generation evaluation, and differs subtly from truthfulness: truthfulness means whether the answer matches facts in the objective world, while faithfulness only requires that the answer does not go beyond the given context; therefore faithfulness is a more measurable and more controllable metric, whereas truthfulness also depends on the reliability of the retrieval source itself.
50%UnverifiedFaithfulness checking is done claim by claim, and this claim-by-claim comparison is more precise than overall scoring, because an answer may have 90% of its content verifiable but mixed with 10% fabricated information, and that 10% hallucination is enough to mislead users into making wrong decisions.
50%UnverifiedRecall@k and MRR are commonly used to evaluate retrieval quality, while evaluating generation typically looks at the consistency between answers and sources, i.e., faithfulness.
50%UnverifiedThe study found that the faithfulness of deletion-based explanations is highly coupled with the detection strength at the time of explanation generation.
50%UnverifiedDeletion testing evaluates explanation faithfulness by masking the regions highlighted by the explanation and observing the degree of drop in model confidence.
50%