RAG Chatbot Pre-Launch Checklist: Six Critical Dimensions

A RAG chatbot isn't just a vector database bolted onto a prompt—it's a complete data pipeline. What truly determines the user experience is whether every stage can withstand testing before launch: source document preparation, retrieval quality, answer grounding, access control, and fallback behavior. The original author, from an AI development company, compiled this first-production-launch checklist based on real delivery experience.

1. Start with a Bounded Use Case

Before writing any code, nail down three things: who uses this assistant, what questions it should answer, and which sources it's allowed to retrieve from.

The original article offers a practical test: "Answer questions about the current product manual" is testable; "Know everything about the company" is not. This sounds like common sense, but a large number of RAG projects die right at the starting line—the use case is defined too broadly, making evaluation impossible.

For the first version, pick just one team and one or two trusted sources. The benefit is a manageable evaluation scope and reduced risk of irrelevant or restricted content leaking into retrieval results. Narrow the boundaries first, then expand later—far more pragmatic than going big from the start.

2. Document Preparation: Subtract Before You Chunk

Before creating embeddings, several cleanup tasks must be completed at the document level:

• Remove duplicate and outdated files—this is the most common source of retrieval noise
• Preserve titles, section names, dates, and source URLs as metadata
• Split by meaningful sections, not purely by character count
• Keep a traceable source reference for each chunk
• Define rules for handling conflicts between two documents

The original article specifically points out that there is no universal correct chunk size. Different chunking strategies need to be tested with questions real users might actually ask. This deserves elaboration: many teams agonize over chunk size for ages but never validate chunking effectiveness with real questions—essentially substituting intuition for evaluation.

Embedding is the process of converting text into high-dimensional numerical vectors, where semantically similar texts are closer together in vector space. Documents must be chunked before being fed into an embedding model because the model has an input length limit, and overly long text makes it hard for a single vector to accurately represent multiple topics. Chunk size is the core parameter at this stage: too small loses context, causing retrieved fragments to be insufficient for a complete answer; too large dilutes semantics and reduces recall precision. A common practice is to use overlapping windows to preserve cross-chunk semantics and validate chunking effectiveness with real questions rather than relying on defaults.

3. Retrieval and Generation Must Be Tested Separately

This is the most methodologically valuable item on the entire checklist.

The approach: build a small evaluation set containing questions close to real scenarios; for each question, annotate "what the correct source passage should be"; first check independently whether the retriever recalls that evidence, then evaluate the model's wording.

The evaluation set should cover several edge cases: paraphrased questions, questions requiring two facts to be combined, queries with typos, and questions that no source can answer. The last category is especially easy to overlook, but it determines whether the system fabricates answers.

One judgment from the original article is worth remembering: a fluent answer generated from the wrong passage is still a failed result. Only by decoupling retrieval quality from language quality in evaluation can you pinpoint which stage the problem actually lies in.

The Retriever is responsible for recalling the most relevant chunks from the vector store for a query, while the Generator is responsible for composing a language answer based on the recalled content. Their failure modes are completely different: retrieval failure manifests as "can't find the correct passage," while generation failure manifests as "the passage is right but the answer is off-topic or fabricated." Retrieval quality is commonly evaluated with Recall@k (whether the top k results contain the correct source) and MRR (the ranking position of the correct source); generation is typically evaluated by answer-source consistency and faithfulness. Merging both sets of metrics into a single score will mask where the real bottleneck is.

4. Require Evidence and Allow a Safe "I Don't Know"

When passing retrieved context to the model, clear instructions are needed: it must answer based on that context, and honestly state when the context is insufficient. Where possible, display the corresponding document or section alongside the answer so users can verify it themselves. For ambiguous or high-impact questions, a channel to escalate to a human must be preserved.

The original article also flags an easy mistake: don't assume the prompt itself can guarantee correctness. A mechanism for reviewing failure cases must be maintained, and the retrieval system or source materials updated accordingly.

5. Access Control and Timeliness Checks

This is the most technically underrated item. A chatbot must never leak document content to users who don't have access—access rules must be enforced at the retrieval stage, not just filtered at the UI layer. Intercepting only at the interface still allows the retrieval layer to recall restricted content, which won't hold up in a security audit.

It's also necessary to define how the index reflects document updates, deletions, and permission changes, and this process must be tested.

For the first release, the original article recommends read-only access—compared to allowing the assistant to modify records or trigger actions, read-only mode has a much smaller control surface.

Enforcing access control at the retrieval layer typically means every chunk in the vector store carries access control metadata (such as department, classification level, user group ID). Queries first filter by the current user's permission scope, then perform vector similarity search. This is called "pre-filtering" or permission-aware approximate nearest neighbor search. Compared to "post-filtering," it's more expensive but safer—the latter may recall restricted content before discarding it, creating leakage risks at the logging or caching level. After permission changes, incremental index rebuilding must also be triggered; otherwise, old permissions persist alongside the vectors.

6. Measure the Real Workflow

Post-launch metrics to track include: whether users can find the correct source, when they rephrase their questions, the frequency of escalations, and whether answers are judged useful. Subject to privacy compliance, sample a portion of real conversations.

The original article mentions an often-confused point: a demo that can answer a few preset prompts and how the tool performs on everyday questions are two different things. Demo questions are hand-picked; production questions are randomly distributed.

A Reusable End-to-End Test

To tie the whole pipeline together, a simple test can serve as a smoke check:

1. Ask a question whose answer actually exists in the source documents
2. Confirm the system recalled the correct passage
3. Confirm the returned answer is supported by that passage
4. Confirm the answer includes a source link
5. Then ask a question with missing evidence and confirm the system refuses to answer or escalates to a human

Repeat this test after changing documents, adjusting chunking, swapping embedding models, or modifying prompts. The original author also linked a Django + FAISS RAG implementation guide covering the complete path from data ingestion to query flow—suitable for teams needing a hands-on reference.

Summary

The core stance of this checklist is: RAG systems need to be validated stage by stage with a pipeline mindset, rather than expecting prompt engineering to solve everything. The six dimensions—use case boundaries, document preparation, independent retrieval evaluation, evidence and fallbacks, access control and timeliness, and workflow measurement—cover the main risk areas from development to launch. For teams preparing their first release, it can be used directly as an actionable checklist framework.

One-line Summary

RAG must be tested stage by stage as a data pipeline before launch, rather than relying on prompt engineering to guarantee correctness.

Paragraph Summary

Based on real delivery experience, this article provides a pre-launch checklist for RAG chatbots heading into first production release. The author argues for treating RAG as a complete data pipeline rather than a simple "vector store plus prompt" assembly. The checklist covers six dimensions: starting with narrowing the use case boundary, then cleanup and metadata preservation before document chunking; emphasizing that retrieval quality and generation quality must be evaluated separately, covering edge cases like paraphrasing, multi-fact combination, typos, and no-answer questions; requiring the model to answer based on evidence and allowing it to say "I don't know"; enforcing access control at the retrieval layer rather than the UI layer, with the index syncing to document updates; and measuring the real workflow after launch. It concludes with a reusable end-to-end smoke test.