RAG Evaluation in Practice: Test Retrieval First, Don't Blame the LLM

When a RAG system produces a wrong answer, most people's first reaction is "the model isn't good enough." But the problem often lies at an earlier stage: the retriever may never have found the correct document, or it may have pushed the key passage far down the list. Evaluating retrieval quality and answer quality together only leads you to repeatedly tune the wrong module.

This article introduces a retrieval-only evaluation tool that uses the minimum amount of code to answer one question: did the system retrieve the expected evidence?

What Retrieval Evaluation Can and Cannot Tell You

There are only two core metrics:

• Recall@k: the proportion of known relevant documents contained in the top-k results. It helps you detect "missing evidence" problems.
• Reciprocal Rank@k: the inverse of the rank of the first relevant result. If it's ranked first, the score is 1; if it's ranked third, the score is 1/3; if it's absent entirely, the score is 0. It helps you distinguish between "evidence ranked first" and "evidence buried below."

Neither metric measures whether the final answer is correct, safe, or grounded. They only answer questions at the retrieval level.

Granularity alignment is an easy trap to fall into: if your retriever returns chunks while annotations are at the document level, you must map chunk IDs back to document IDs before scoring, otherwise the metrics will be distorted.

The trade-off between Recall@k and Precision@k is often confused. Recall@k focuses on "was what should have been found actually found," while Precision@k focuses on "of what was found, how much is relevant." Retrieval evaluation prioritizes Recall, because missing evidence leaves downstream generation with no way to recover, whereas bringing back a small amount of noise can usually be filtered out by the context window or a reranker.

MRR (Mean Reciprocal Rank) is the average of Reciprocal Rank across all answerable cases. A single case's RR only takes the position of the first relevant result, so it naturally rewards only "the earliest hit" and doesn't care whether the other relevant documents were also recalled. When a question requires multiple pieces of evidence to be assembled, MRR may overestimate retrieval quality; in that case, you should also look at Recall@k or introduce metrics like nDCG that reflect the overall quality of the ranking.

Granularity refers to the alignment level between the retrieval return unit and the annotation unit—document, chunk, paragraph, or sentence. Cross-level comparisons must be mapped first, otherwise a single chunk hit will be counted as an entire document hit, inflating the metrics.

Building Reviewed Test Cases

Each answerable question needs a set of expected document IDs. Cases should come from typical questions in real workflows, and a human review step is mandatory.

For unanswerable questions, an empty relevant document set does not mean "any retrieved document is irrelevant." Its meaning is: no known answer exists in the approved corpus. Such cases should be reported separately and excluded from recall and ranking average calculations.

A Runnable Deterministic Example

The code below uses only the Python standard library. Replace the fixture retrieval function with your own retrieval API to use it for actual testing.

from dataclasses import dataclass
from statistics import mean
from typing import Callable, Sequence

@dataclass(frozen=True)
class Case:
    case_id: str
    question: str
    relevant_ids: frozenset[str]


def score(case: Case, retrieved: Sequence[str], k: int) -> dict:
    if k < 1:
        raise ValueError("k must be positive")
    top = list(dict.fromkeys(retrieved))[:k]
    relevant = case.relevant_ids

    if not relevant:
        return {"case_id": case.case_id, "answerable": False,
                "retrieved_ids": top, "recall": None, "rr": None}

    hits = relevant.intersection(top)
    first_hit = next((rank for rank, doc_id in enumerate(top, 1)
                      if doc_id in relevant), None)
    return {"case_id": case.case_id, "answerable": True,
            "retrieved_ids": top,
            "recall": len(hits) / len(relevant),
            "rr": 1 / first_hit if first_hit else 0.0}

The evaluate function iterates over all cases, calls score, and then aggregates mean_recall, MRR for answerable cases, and the counts of answerable and unanswerable cases.

The example data is designed with three groups of cases: a refund question expected to hit refund-policy, a cancel question expected to hit cancel-guide, and an unanswerable question expected to have an empty set. The fixture retrieval results deliberately rank the correct document second in the refund scenario and completely omit the correct document in the cancel scenario.

Run results: mean_recall is 0.5, MRR is 0.25, and the number of unanswerable cases is 1. These averages only describe these two answerable fixture cases and are not an estimate of production performance.

A key detail: dict.fromkeys deduplicates while preserving rank, because duplicate documents should not occupy multiple rank positions.

Check Failure Causes Before Modifying the System

For each failed case, troubleshoot in order:

1. Is the expected document in the indexed corpus?
2. After the corpus version is updated, is the annotation still correct?
3. Has the relevant text been split across different chunks?
4. Has a metadata filter excluded it?
5. Does the query use terminology that the embedding model cannot match?
6. Has reranking pushed the evidence out of the context budget?

Change only one component at a time, and run the same reviewed cases against the same corpus snapshot. Keep a held-out set—repeatedly tuning parameters on the evaluation set can make the scores look good without representing any real improvement in the actual workflow.

Items 5 and 6 in the troubleshooting checklist involve two components that are often underestimated. An embedding model maps queries and documents into the same vector space and matches them using measures such as cosine similarity; when a query uses domain abbreviations, internal codenames, or wording that differs significantly from the training corpus distribution, vector distance may fail to reflect semantic relevance, causing the correct document to rank far down or not appear in the candidate set at all.

Reranking is the process of reordering candidates after initial recall using a heavier but more precise model (commonly a cross-encoder, which concatenates the query and document and scores them jointly). It can significantly improve ranking quality, but it consumes additional context budget—if the final top-n passed to the generation model is too small and the reranker pushes the correct evidence past position n, it looks like a ranking problem on the surface but is actually a budget configuration problem. Item 6 is precisely about checking this boundary.

Answer Evaluation Is an Independent Second Stage

After retrieval passes, test the generated answer with explicit scoring criteria:

• Are factual statements supported by the supplied passages?
• Do citations point to evidence that supports the corresponding statements?
• Does it answer the user's actual question?
• When evidence is insufficient, does it ask for clarification or refuse to answer?
• Does it comply with access controls and avoid exposing restricted data?

Automated checks and model judging can help with initial screening, but their judgments still need human review and verification. Passing retrieval tests does not mean the answer is correct or authorization-safe.

Keep Evidence Reproducible

Record case IDs, corpus/index versions, retrieval settings, model versions, prompt versions, and retrieved IDs. Pay attention to permissions when storing passages and outputs, and desensitize tracking records before sharing them.

Production comparisons require more representative cases, repeated tests of stochastic runs, reporting denominators and sampling limitations, and tracking latency and cost in addition to quality.

Practical conclusion: first locate which stage the failure occurs in. Prompt rewriting cannot recover evidence that the system never retrieved.

One-line Summary

Evaluate the RAG retrieval stage separately with Recall@k and MRR; locate the failure stage before tuning.

Paragraph Summary

The article argues for splitting RAG evaluation into two independent stages, retrieval and generation, to avoid broadly attributing answer errors to the model. Retrieval evaluation needs only two metrics: Recall@k and MRR—the former detects missing evidence, and the latter distinguishes whether evidence ranks first or is buried. Test cases must come from real workflows and undergo human review; chunk and document granularity must be aligned, and unanswerable questions should be reported separately rather than included in averages. The article provides a deterministic example using only the Python standard library, demonstrating how to score deduplicated results and aggregate mean_recall and MRR. Troubleshooting proceeds along six directions: corpus, annotations, chunk splitting, metadata filtering, embedding matching, and reranking—changing only one variable at a time and keeping a held-out set. After retrieval passes, move to independent answer evaluation, and ultimately record version information to ensure reproducibility.