7 Evaluation Dimensions to Test Before Launching RAG: A Practical Checklist

Why a Successful RAG Demo Doesn't Mean Production Reliability

In LLM application development, a common cognitive trap is treating a successful demo as proof of system reliability. A RAG chatbot might answer ten questions correctly in a row, but the moment a user asks about an outdated policy, a document missing from the knowledge base, or information that requires integrating multiple sources, its fatal flaws surface.

According to developer Taha Hussein's experience shared on his technical blog, RAG evaluation should not be a one-off scoring exercise, but an engineering feedback loop—helping you discover failures, understand their causes, and make well-grounded improvements. He proposed an evaluation checklist with 7 dimensions, covering the entire chain from retrieval to generation to reproducibility.

The core logic of this checklist: before considering whether a system is ready for broader testing, first validate it with a minimal but complete evaluation framework.

Retrieval Quality: Can It Find It, and Does It Rank It High?

1. Can retrieval find the right documents?

The first step in evaluation is to prepare a set of questions and manually label the relevant document IDs for each. Then compute Recall@K—the proportion of relevant documents recalled among the retriever's top K results.

Hussein provides a concise Python implementation, assuming the retrieval results contain a document ID field. For questions that require multiple documents to answer, all relevant documents must be recorded—retrieving only one useful snippet may not be enough to answer questions requiring multiple pieces of evidence.

The key here is that retrieval evaluation must be independent of the generation stage. If retrieval itself cannot find the right documents, no amount of generation strength will help.

Recall@K (Recall at K) is the core metric for measuring a retrieval system's coverage capability, defined as: among the top K results returned by the retriever, the proportion of relevant documents recalled out of all relevant documents in the test set. For example, if a question has 3 relevant documents and the retriever hits 2 among its top 5 results, then Recall@5 = 2/3 ≈ 0.67.

The choice of K should be determined by the actual context window of the downstream generation stage—if only the top 5 chunks are concatenated during generation, then evaluating Recall@5 is more meaningful than Recall@20. Recall@K is sensitive to "misses" but completely indifferent to where relevant documents rank, so it must be used together with ranking metrics (Precision@K, MRR, NDCG@K) to fully characterize retrieval quality. In RAG scenarios, Recall usually takes priority over Precision: missing a key piece of evidence often directly leads to a wrong answer, while recalling a few extra irrelevant documents merely adds context noise and token cost.

2. Are relevant results ranked at the top?

Finding the right document is only the first step; its ranking position is equally critical. If a relevant document appears at position 20, it likely won't make it into the generation model's context window at all.

Hussein suggests evaluating three ranking metrics: Precision@K (the proportion of relevant documents among the top K results), MRR (Mean Reciprocal Rank), and NDCG@K (Normalized Discounted Cumulative Gain). These metrics help you determine whether the retriever effectively ranks useful information at the top.

If the problem lies in ranking, potential improvement directions include hybrid retrieval, metadata filtering, and reranking.

MRR (Mean Reciprocal Rank) measures "how early the first relevant result appears": for each question, take the reciprocal of the rank of the first relevant document (rank 1 scores 1.0, rank 2 scores 0.5, rank 3 scores 0.33...), then average across all questions. MRR is extremely sensitive to the "first relevant result," making it suitable for question-answering scenarios, but it completely ignores other relevant documents after the first.

NDCG@K (Normalized Discounted Cumulative Gain) is more comprehensive: it assigns a relevance score to each document at each position (typically 0/1, but can be graded), then applies a logarithmic discount based on ranking position—the lower the rank, the smaller the contribution—and finally normalizes by dividing by the score under the ideal ranking, so results fall between 0 and 1. NDCG@K considers both relevance strength and ranking position, making it one of the most powerful comprehensive metrics in retrieval evaluation today. The division of labor among the three is roughly: Precision@K looks at "how many in the top K are relevant," MRR looks at "how high the first relevant one ranks," and NDCG@K looks at "how far the overall ranking is from the ideal ranking."

Generation Quality: Faithful to Context, Not to the Real World

3. Is the generated answer faithful to the retrieved context?

Hussein gives an intuitive example: the retrieved context says "customers can request a refund within 30 days of purchase," but the model generates the answer "customers can request a refund within 90 days." The answer directly contradicts the provided evidence.

Faithfulness evaluation helps detect such unsupported claims. But here's an easily confused point: faithfulness does not equal correctness. The model can faithfully repeat an outdated or incorrect document. For important applications, both the source material and the generated answer must be verified.

Faithfulness and Correctness are the pair of concepts most easily conflated in RAG evaluation. Faithfulness measures whether the generated answer "only says what the context supports," which is an internal consistency check—it doesn't care whether the context itself is right. Correctness measures whether the answer "matches real-world facts" or a human-annotated reference answer, which is an external consistency check.

The relationship between the two is one-directional: faithful doesn't necessarily mean correct (faithfully repeating an outdated policy), and correct doesn't necessarily mean faithful (answering correctly by chance but citing irrelevant evidence). Therefore, a mature evaluation system needs to test both simultaneously, and additionally audit the quality of the knowledge base source material itself. In practice, faithfulness is usually automatically determined using LLM-as-a-Judge or NLI (Natural Language Inference)-type models, breaking the answer into atomic claims and checking one by one whether they are entailed by the context; correctness relies on human-annotated reference answers or authoritative fact sources. Confusing the two leads to misjudgments—for example, treating "the model faithfully cited a wrong document" as a pass, or treating "the model answered correctly from parametric knowledge without citing evidence" as qualified.

4. Can the system handle missing information?

A reliable RAG system should not fabricate answers when the knowledge base doesn't contain the required information. Hussein suggests adding such test cases to the test set:

{
    "question": "Does the company support product X?",
    "expected_behavior": (
        "State that existing documentation doesn't provide enough information."
    ),
}

Test whether the system acknowledges uncertainty and avoids unsupported claims. This is especially important for internal knowledge assistants, customer support, and policy-related applications—in these scenarios, a confidently wrong answer can have real consequences.

Complex Scenarios and Continuous Validation

5. Can it answer questions that require multiple documents?

Simple questions are not enough to test the true capability of a RAG system. You need to create test cases whose answers depend on multiple documents or sections.

For example, one document contains product eligibility requirements, another contains related exceptions, and a third contains effective dates. Check whether retrieval captures all necessary evidence, and whether the final answer correctly combines this information. Such cases often reveal weaknesses that simple demos mask.

6. Does performance remain stable after changes?

Suppose you swap the embedding model or increase the chunk size. The new version may improve performance on some questions while breaking others.

The key practice: run both versions on the same evaluation dataset simultaneously, comparing retrieval metrics, answer correctness, faithfulness, latency, token usage, and cost. Save the configuration for every run. Don't assume a change is an improvement just because a few manually tested answers look better.

7. Can failures be traced and reproduced?

For each evaluation case, record enough information to investigate what happened: the question and expected answer, retrieved document IDs and chunks, similarity scores, prompt and model configuration, generated answer, metric results, latency, and token usage.

Hussein particularly cautions: avoid unnecessarily logging keys or sensitive user data. These records make debugging much easier—you can distinguish whether it's a retrieval problem or a generation problem, rather than guessing.

Minimal RAG Evaluation Checklist

Before considering advancing the system to broader testing, Hussein suggests confirming item by item:

• A representative test dataset exists
• Relevant documents have been manually verified
• Retrieval is evaluated independently
• Answers are compared against reference data
• Questions about missing information are included
• Changes are tested on the same dataset
• Failures can be traced and reproduced

Passing this checklist does not guarantee production readiness. Security, access control, privacy, load testing, and domain-specific requirements still need separate validation. But it provides a much stronger foundation than "ask the chatbot a few questions and judge whether it works."

Evaluation Is a Feedback Loop, Not a Scoring Exercise

Hussein concludes at the end of the article: RAG evaluation is not just a one-time score, but an engineering feedback loop. It helps you discover failures, understand their causes, and make well-grounded improvements. These seven dimensions—from retrieval coverage to generation faithfulness, from complex scenarios to change regression—form a closed loop, where each link provides debuggable signals for the next.

For teams building RAG applications, the value of this checklist lies in transforming "it feels like it works" into "knowing where it might not work."

One-line Summary

RAG evaluation should be an engineering feedback loop from retrieval to generation, not a scoring exercise where passing the demo means going live.

Paragraph Summary

The article points out that a RAG chatbot's streak of correct answers during the demo phase does not prove system reliability in production—flaws surface as soon as it encounters outdated policies, missing documents, or questions requiring cross-source integration. Developer Taha Hussein proposes an evaluation checklist with 7 dimensions: at the retrieval quality level, independently evaluate recall metrics like Recall@K, as well as ranking metrics like Precision@K, MRR, and NDCG@K; at the generation quality level, detect whether answers are faithful to the retrieved context (noting that faithfulness is not the same as real-world correctness), and test whether the system can acknowledge uncertainty rather than fabricate when information is missing from the knowledge base; at the complex scenario level, cover multi-document question answering, post-change regression comparison (running old and new versions on the same dataset, comparing metrics and costs), and traceable, reproducible failures. The article concludes by emphasizing that evaluation is a closed loop for discovering failures, understanding causes, and making well-grounded improvements—passing the checklist does not equal production readiness, and security, permissions, privacy, and load testing still require separate validation.