RAG Evaluation
A systematic evaluation methodology for Retrieval-Augmented Generation (RAG) systems. It uses quantitative metrics to measure the quality of multiple pipeline stages, including retrieval recall and generation accuracy, helping teams identify bottlenecks and guide optimization efforts.
Timeline (last 90 days)
Large model evaluation, RAG evaluation and agent evaluation are three problems at different levels and cannot be handled with the same set of methodologies.
Avoid questions like 'summarize this document'; instead use adversarial retrieval questions (mutually contradictory clauses, exceptions buried in appendices, numbers that appear only in tables) to evaluate RAG retrieval quality.
The core of RAG evaluation is judging whether what the system "retrieves" and the "answers it generates" truly satisfy user needs, usually comprising two layers: retrieval quality and generation quality.
Generation faithfulness, context relevance, and answer relevance together form the classic triangle framework for RAG evaluation.
If the answer usually falls in a single text chunk, priority should be given to MRR and Hit rate; if the answer spans multiple text chunks, priority should be given to Recall and nDCG.
A fluent answer generated based on the wrong passage is still a failed result.
Building a small test set should start with 30 to 50 real questions, each accompanied by the correct answer and the document containing the answer, measuring retrieval (whether the correct text chunk appears in the top results) and generation (whether the final answer is supported by the retrieved text) separately.
The primary goal of evaluation is accuracy, not performance.
The programmatic evaluation process includes generating a question set and reference answers, driving the RAG system to produce actual answers, and scoring by comparing the actual answers against the reference answers; it is best to perform manual verification before running the tests.
Automated evaluation forms a two-tier optimization mechanism: programmatic evaluation serves as an initial filter by developers (a basic threshold for launch), while manual evaluation serves as the final review by business/test personnel (final refinement close to real users).
All Facts (10)
Large model evaluation, RAG evaluation and agent evaluation are three problems at different levels and cannot be handled with the same set of methodologies.
50%UnverifiedAvoid questions like 'summarize this document'; instead use adversarial retrieval questions (mutually contradictory clauses, exceptions buried in appendices, numbers that appear only in tables) to evaluate RAG retrieval quality.
50%UnverifiedThe core of RAG evaluation is judging whether what the system "retrieves" and the "answers it generates" truly satisfy user needs, usually comprising two layers: retrieval quality and generation quality.
50%UnverifiedGeneration faithfulness, context relevance, and answer relevance together form the classic triangle framework for RAG evaluation.
50%UnverifiedIf the answer usually falls in a single text chunk, priority should be given to MRR and Hit rate; if the answer spans multiple text chunks, priority should be given to Recall and nDCG.
50%UnverifiedA fluent answer generated based on the wrong passage is still a failed result.
50%UnverifiedBuilding a small test set should start with 30 to 50 real questions, each accompanied by the correct answer and the document containing the answer, measuring retrieval (whether the correct text chunk appears in the top results) and generation (whether the final answer is supported by the retrieved text) separately.
50%UnverifiedThe primary goal of evaluation is accuracy, not performance.
50%UnverifiedThe programmatic evaluation process includes generating a question set and reference answers, driving the RAG system to produce actual answers, and scoring by comparing the actual answers against the reference answers; it is best to perform manual verification before running the tests.
50%UnverifiedAutomated evaluation forms a two-tier optimization mechanism: programmatic evaluation serves as an initial filter by developers (a basic threshold for launch), while manual evaluation serves as the final review by business/test personnel (final refinement close to real users).
50%