Building More Reliable RAG Systems: Four Core Levers for Retrieval Optimization

Most RAG (Retrieval-Augmented Generation) systems that disappoint in production don't fail at the generation stage — they fail at retrieval. The model produces fluent-sounding answers based on the wrong context, or has no context to work with at all. Building a demo that runs takes an afternoon; making a system work reliably with real queries, real documents, and real traffic requires deliberate engineering at every stage of the pipeline.

This article focuses on four key levers that determine whether a RAG system succeeds or fails: pre- and post-retrieval processing strategies, document chunking, embedding model and vector store tuning, and how to measure whether any of it is actually working. Together they form a closed loop — evaluation tells you which lever to fix next.

1. Pre- and Post-Retrieval Processing

User queries tend to be short, vague, and phrased very differently from the documents that could answer them. Pre-retrieval processing bridges that gap before touching the index, while post-retrieval validation intercepts poor context before it reaches the model.

Before Retrieval

Query rewriting is the first step: use an LLM to transform an ambiguous, conversational question into a standalone, retrieval-friendly expression. In conversational settings, this also means resolving references like "what about the second one?" using conversation history.

Query expansion and multi-query generate multiple paraphrase variants, retrieve against each separately, then merge the results — this significantly improves recall when users and documents use different vocabulary. HyDE (Hypothetical Document Embeddings) takes a different approach: have the model write a plausible-sounding answer first, then embed that answer rather than the original question, because answers tend to sit closer to real documents in embedding space than questions do.

Other techniques include query decomposition (breaking "compare the pricing and limits of plans A and B" into sub-queries), routing (directing queries to specific indexes, SQL databases, or web search — or skipping retrieval entirely for questions the model can answer directly), and metadata filtering (extracting constraints like date, product, or language from the query to use as filters). Skipping unnecessary retrieval reduces both latency and model distraction.

HyDE (Hypothetical Document Embeddings) is worth unpacking. In embedding space, short questions like "What is quantum entanglement?" are naturally distributed differently from long, knowledge-dense paragraphs — they lack domain vocabulary and contextual framing, which leads to lower cosine similarity scores. HyDE has the LLM generate a "hypothetical answer" from scratch. This text is closer in vocabulary density and stylistic register to real documents in the corpus, so its embedding lands in a more favorable retrieval region. Even if the generated answer is factually imperfect, the semantic neighborhood of its embedding vector often still surfaces the correct real documents. The tradeoff is one extra LLM call per retrieval, making it best suited for latency-tolerant, recall-critical scenarios.

After Retrieval

Reranking is typically the highest-return intervention per unit of effort: retrieve a larger candidate set (say, 30–50 results), then use a cross-encoder or dedicated reranking model to score each passage jointly with the query for a more accurate relevance estimate.

Relevance validation introduces a lightweight scorer to judge whether retrieved passages are actually on-topic — if not, rewrite the query and retry, or switch data sources. This is the core idea behind corrective approaches like CRAG. Combined with context compression (trimming passages down to the truly critical sentences) and post-generation grounding checks (requiring every claim to be supported by retrieved text with citations, and returning "I don't know" otherwise), the trustworthiness of the entire pipeline improves substantially.

It's worth emphasizing: every step adds latency and cost, so add them one at a time in response to observed failure modes rather than enabling everything by default.

Cross-encoders differ architecturally from the bi-encoders used for initial retrieval. Bi-encoders encode the query and document independently and then compute similarity — fast, but limited in precision. Cross-encoders concatenate the query and candidate passage and feed them through the model together, allowing attention to perform fine-grained interaction across both texts for a more accurate relevance judgment. The downside: document vectors can't be precomputed offline, so scoring must happen at query time. This makes cross-encoders suitable only for reranking a narrowed candidate set (typically 20–50 results), not for scanning the full index. Popular open-source rerankers include Cohere Rerank, BGE-Reranker, and the ms-marco series — all capable of scoring a batch in millisecond-range latency.

2. Chunking Strategy

Chunking defines what a "unit of knowledge" looks like to the retriever. Chunks that are too large bury relevant sentences in noise and dilute the embedding; chunks that are too small lose the context needed to make the content interpretable. There is no universally optimal size — the goal is to match chunking to the actual structure of the content and how it will be queried.

Common strategies include: fixed-size (split by N tokens — good for quick baselines), recursive splitting (split by paragraph, then sentence, then word — a solid default for general prose), structure-aware (split on heading, section, code block, or table boundaries — suited for documentation and legal text), semantic chunking (start a new chunk where embedding similarity between adjacent sentences drops), parent-child chunking (embed small chunks but return the larger parent section), and sentence windows (embed individual sentences, return surrounding neighbors when one is hit).

Practical Guidance

Start simple: recursive splitting at roughly 300–800 tokens with 10–15% overlap is a reasonable baseline, then tune using an evaluation set rather than intuition. Always respect structure — never split a table, code block, or numbered list in the middle, and keep headings attached to the text beneath them.

One frequently overlooked but critical technique is adding context to chunks: prepend the document title and section path to each chunk before embedding, or use an LLM to generate a short summary explaining where the chunk sits in the document (the so-called "contextual retrieval" approach). A chunk that reads "it grew by 12%" is worthless without knowing what "it" refers to. You should also store rich metadata, handle non-text content explicitly (convert tables to text preserving row/column structure, run OCR or generate captions for images), and version your chunking scheme — every change to chunking logic requires re-embedding.

3. Vector Store and Embedding Optimization

Even with perfect chunking, retrieval quality is bounded by whether the embeddings capture semantics in your domain and whether the index can find nearest neighbors at your scale.

Choosing Embeddings

Choose models based on your data, not leaderboard rankings: public benchmarks are a starting point only. Always test two or three candidate models on your own queries and documents — domain-specific vocabulary in legal, medical, code, or multilingual settings can completely change the rankings. Balance dimensionality against cost — higher-dimensional vectors are generally more expressive but increase storage, memory, and search latency. Models that support truncation (Matryoshka-style embeddings) let you reduce vector size with minimal quality loss.

Also pay attention to using the correct input format (some models require different prefixes for queries versus documents), consider fine-tuning (labeled query-passage pairs can substantially improve retrieval in specialized domains; when you lack data, synthetic queries can bootstrap training), and maintain consistent normalization (distance metrics must match how the model was trained).

Vector Store and Index Tuning

Choose your index based on scale: exact (flat) search is fine for small collections; larger collections need approximate nearest-neighbor indexes like HNSW or IVF, which trade a small amount of recall for large speed gains — tune parameters like ef_search and M against recall targets on your own data. When memory is a constraint, quantization (scalar or product quantization) can dramatically compress the index with typically small recall loss.

Hybrid search deserves early adoption: combine dense vector search with BM25 keyword search — the former handles semantic meaning and paraphrases, the latter handles exact terms, product codes, names, and rare jargon — then merge with Reciprocal Rank Fusion (RRF) or weighted scores before reranking. Also filter efficiently (index filterable fields and understand whether your vector store filters before or after ANN search), plan for data freshness (how updates, deletes, and re-embeddings after a model change propagate — use aliases or versioned indexes for zero-downtime cutover), and watch operational metrics — query latency percentiles (p50/p95), index build time, memory footprint, and per-query cost. Even the most accurate system can't ship if it's too slow or too expensive.

Reciprocal Rank Fusion (RRF) is the most widely used parameter-free method for merging results from multiple retrieval sources. The core formula: for each document, sum its reciprocal rank position (plus a smoothing constant k, typically 60) across all result lists to produce a final score. This operates on rank positions rather than raw similarity scores, making it naturally compatible with signals from different measurement scales — cosine similarity from dense vectors and TF-IDF scores from BM25 don't need to be aligned to the same range. In practice, RRF is robust across most tasks and requires no dataset-specific weight tuning, making it a low-cost starting point for hybrid search. For further improvement, layer a reranking model on top of RRF, or learn fusion weights for each retrieval signal.

4. Evaluating RAG Performance

You can't improve what you can't measure. For RAG, you must measure two things separately: whether retrieval found the right material, and whether generation used it well. Conflating the two makes failures impossible to diagnose.

Evaluation should cover four dimensions: retrieval (did it return passages containing the answer — measured with Recall@k, Precision@k, MRR, nDCG), generation (is the answer correct and grounded — measured with faithfulness, answer relevance, answer correctness), context utilization (was only the necessary content retrieved — measured with context precision and recall), and system (is it production-ready — latency p50/p95, per-query cost, abstention rate).

Building a Golden Dataset

Start with 50–200 real questions, each paired with the source passages that answer it and ideally a reference answer. Source questions from real user logs when possible. Cover simple lookups, multi-hop reasoning, ambiguous queries, and questions that have no answer in the corpus — a good system should honestly say "I don't know." Synthesizing questions from documents is a fast way to scale up the dataset, but always run a human spot-check on a sample.

Multi-hop reasoning queries are an indispensable hard case in any golden dataset. These questions can't be answered from a single passage — they require chaining inference across two or more document fragments. For example: "What university did the engineer leading that project attend?" — first you need to find the engineer's name, then look up their educational background in a different passage. Standard single-round retrieval has low success rates on these queries, exposing the system's limits in multi-step reasoning, iterative retrieval, and context stitching. If you exclude this type of question from your evaluation set, scores will be inflated — and when real users ask compound questions in production, you'll see a sharp performance cliff.

Evaluating Generation Quality Carefully

LLM-as-judge is a practical method for scoring faithfulness and relevance at scale: give the judge clear rubrics, the question, context, and answer, and ask for a score with a brief rationale. But you must calibrate the judge — compare its ratings against human annotations on a sample, watch for its tendency to favor longer or more confident-sounding answers, and re-validate when switching judge models. Frameworks like RAGAS, TruLens, DeepEval, and LangSmith have already implemented these metrics, but understand what a number actually measures before trusting it.

Running It as a Closed Loop

Establish a baseline on the golden set before changing anything. Change only one variable at a time (chunk size, embedding model, reranker) and rerun. Break down results by query type, because averages hide categories that are badly broken. Read failure cases manually one by one, classifying them as retrieval misses, ranking problems, context issues, or generation problems, then fix the component that's actually responsible. Gate deployments in CI using the evaluation set to catch regressions. Monitor continuously in production and feed hard cases back into the golden set.

A Sensible Order of Operations

A pragmatic sequence: build the evaluation set first, then get a simple baseline with recursive chunking and a strong embedding model, then add hybrid search and reranking (they tend to pay off early), and only then tune chunking and add query-side techniques where failure analysis points. Every change should earn its place with a measurable improvement.

RAG quality comes from the entire pipeline working together, and the teams that do it well are the ones who can determine, query by query, exactly which stage let the user down.