The Dividing Line Between RAG Systems and Demos: Evaluation Gates and Golden Sets

Most RAG projects stop at the demo stage: you ask a question, the system retrieves snippets, and the model generates a plausible-looking answer. The audience nods, but nobody can say for sure whether the retrieval results actually contained the correct answer. The problem isn't model hallucination—it's that retrieval quality quietly degrades with no one monitoring it. Jordi Marsal, author of the open-source project CodebaseRAG, offers a simple criterion: the distance between a RAG demo and a RAG system is a golden set plus a CI gate.

The Problem: Nobody Measures Recall

The typical lifecycle of a RAG demo goes like this: the initial version works well, then someone tweaks the chunking strategy, swaps the embedding model, or migrates to a new vector database. These changes may improve or harm retrieval quality, but the team's only signal is "the answers feel worse." This vague perception can't pinpoint the problem, let alone prevent regressions.

The author calls this failure mode "unobserved retrieval degradation." A demo is optimized for the success path in front of an audience; a system is optimized for a number that can be compared across regressions. Without evaluation, there's no baseline to regress against; without a baseline, every change is a gamble.

recall@5 is a common metric in retrieval evaluation, defined as: for each query, check whether the top 5 retrieved results contain at least one relevant document (or the proportion of all relevant documents that are recalled), then average across all queries. In a golden set scenario, each question typically maps to one or more ground-truth answer locations, and recall@5 measures "does the correct answer appear in the top 5 results?" It doesn't care about precise ranking—only whether the answer was found—so it's highly sensitive to the retrieval system's "floor capability." Compared with precision or MRR, recall@5 is more tolerant of ranking noise, making it suitable as a hard threshold for a gate: once recall drops, it means the system is starting to miss correct answers, which is a user-perceivable degradation signal.

The Golden Set: Turning "Correct" into Forty Checkable Q&A Pairs

The honesty of an evaluation depends on the quality of the questions. CodebaseRAG's golden set is a public list of Q&A pairs, where each question is one a developer would actually ask and each answer is annotated with its location in the codebase. The set contains at least 40 pairs—enough to catch real regressions, yet small enough that every pair can be manually reviewed.

No annotation platform is needed—just a place to record questions and the discipline to maintain them. These Q&A pairs live next to the code, so changes to retrieval logic and their corresponding expectation updates flow through the same review.

For example, the golden set contains entries like: which abstraction does the query use case depend on, and why isn't it a vector store type? The answer is the retriever port; adapters implement that port, and core code never imports pgvector or Qdrant classes. Another example: what should CI do when average recall@5 falls below the committed baseline? The answer is to fail the build; the baseline is a committed number, and moving it requires an explicit ADR decision—not a casual edit to make CI green.

These examples aren't hard; their purpose is to make answers defensible, so that a wrong answer becomes a failing test rather than an argument.

The CI Gate: What Fails and What Doesn't

A gate is only useful if it fails loudly and predictably. CodebaseRAG's gate measures average recall@5, mean reciprocal rank (MRR), and nDCG@5 on the golden set, and the build fails when average recall@5 drops below the committed baseline of 0.409. The rule is that simple. CI runs the same deterministic evaluation as the notebook, so a red build points to a retrieval change, not environment flakiness.

What doesn't fail matters just as much. Model changes pass as long as recall@5 stays above the baseline, even if MRR and nDCG@5 fluctuate—the gate guards the promised metric, not every number on the dashboard. Refactors that don't change retrieval logic pass. The gate doesn't judge answer fluency: this is retrieval evaluation, and pretending it measures generation quality is as dishonest as having no evaluation at all.

MRR (Mean Reciprocal Rank) and nDCG@5 (normalized discounted cumulative gain@5) are two ranking-sensitive metrics. MRR focuses on where the first relevant result appears: if the first relevant result is ranked 1st, the reciprocal is 1; 2nd is 1/2, and so on, averaged across all queries. It penalizes "correct answers appearing late." nDCG@5 considers both the relevance grade and positional discount of the top 5 results—more relevant and higher-ranked yields a higher score—then normalizes by dividing by the score under ideal ranking. Combined with recall@5, these two provide a fuller picture of retrieval quality: recall@5 checks "whether it's there," MRR checks "how quickly it's found," and nDCG@5 checks "how good the overall ranking is." The gate uses only recall@5 as a hard threshold to avoid build instability caused by ranking fluctuations, while keeping MRR and nDCG@5 as observational metrics.

Hexagonal Architecture: Swapping Vector Stores Must Not Rewrite the Core

Since what's being measured is retrieval, the object being measured can't be welded to the deployment method. CodebaseRAG adopts a hexagonal architecture: the query use case depends on ports, with pgvector, Qdrant, Ollama, and Anthropic sitting behind those ports as adapters. The test is concrete: swapping Qdrant for pgvector, or running fully local with Ollama, must not require rewriting the core. If changing the vector store forces changes to use case code, then the evaluation is measuring an incidental property of the storage layer, and every future swap reopens the question of whether the metrics are still meaningful.

Keeping the core independent of frameworks and transport layers brings more than cleanliness: it keeps evaluation deterministic, running against in-memory or container-backed adapters with no network. A committed baseline is only trustworthy if the underlying code doesn't deform as infrastructure changes.

Hexagonal architecture, also known as the ports-and-adapters pattern, was introduced by Alistair Cockburn. Its core idea is to isolate an application's core business logic from external technical infrastructure: the core expresses what capabilities it needs by defining "ports" (interfaces), while "adapters" implement those ports and connect to concrete technologies such as databases, message queues, or web frameworks. This way, swapping external technology only requires replacing the adapter, with no changes to core code. In a RAG system, the query use case depends on a "retriever port," and vector stores like pgvector and Qdrant implement that port as adapters. This design lets evaluation run against in-memory fake adapters, ensuring determinism and speed, while keeping baseline numbers comparable across infrastructure changes.

The Current Baseline and What It Doesn't Claim

The committed baseline is an average recall@5 of 0.409 (MRR 0.231, nDCG@5 0.277) on a golden set of at least 40 pairs. Read it as a floor, not a result. 0.409 is not SOTA, nor does it claim the retrieval problem is solved. It's the number the project commits to defending, chosen low enough to stay stable and high enough to catch real declines.

This number doesn't claim: that the golden set is exhaustive, that a green build means good answers, or that the embedding model is optimal. Raising the baseline is a documented decision with a rationale, not a vanity metric pasted into the README.

What it does claim is smaller but more useful: on every change, the repository can tell you whether retrieval got worse, and prove it from a public golden set. That's the dividing line between a RAG demo and a RAG system—a demo gives you an answer; a system lets you disagree with it.

One-line Summary

RAG demos vs. systems: a golden set plus a CI gate that uses regressable retrieval metrics to prevent silent degradation.

Paragraph Summary

The article argues that most RAG projects stop at the demo stage, with the core problem being silent degradation of retrieval quality under no monitoring. CodebaseRAG's author proposes distinguishing demos from systems with a public golden set of at least 40 pairs plus a CI gate: the gate measures average recall@5, MRR, and nDCG@5, uses recall@5 ≥ 0.409 as a hard baseline, and guards only the promised metric to avoid over-constraint. Hexagonal architecture ensures swapping vector stores or embedding models doesn't rewrite the core, keeping evaluation deterministic and comparable. The 0.409 baseline is explicitly a floor, not SOTA, and raising it requires an explicit ADR decision. The bottom line: a system lets you disagree with retrieval results, while a demo just gives you an answer.