RAG Retrieval Optimization in Practice: Hybrid Search, Reranking, and Recall Evaluation

RAG systems give wrong answers, and the problem is often not the generation model itself. The model simply answers based on the retrieved text chunks—if the correct chunk was never retrieved in the first place, even the strongest model can't help. Fix the retrieval stage, and most wrong answers will disappear on their own.

First, Measure Retrieval Quality in Isolation

Before tweaking prompts or swapping models, you need to establish a retrieval evaluation baseline. Prepare 50 to 100 real questions and label each one with the text chunk that can actually answer it. Then measure retrieval performance in isolation, without the generation model.

There are two core metrics:

• recall@k: The proportion of questions where the top k results contain at least one correct text chunk. If recall@5 is only 60%, that means four out of ten questions have no answer available before the model even starts writing.
• MRR (Mean Reciprocal Rank): The average of the reciprocal ranks of the first correct text chunk. It reflects whether correct results tend to appear near the top, or barely squeeze into 9th place.

Every subsequent change should move these two numbers. If the metrics don't budge, the change had no real effect.

Split by Document Structure, Not Character Count

Fixed 500-character splitting cuts tables in half and separates step 4 from steps 1–3. A better approach is to split according to the document's own structure: keep headings, paragraphs, list items, and table rows intact.

Each text chunk should carry a heading path prefix, such as "Billing > Refunds > Partial Refunds," so that even if the chunk only says "this takes 5 days," it's clear what "this" refers to.

Chunk size is a tradeoff. Chunks that are too small lose context; chunks that are too large mix multiple topics into a single embedding vector, diluting the key sentence.

Hybrid Search: Combining Keywords and Vectors

Embeddings excel at semantic matching but perform poorly on exact strings. When a query involves error code E1042, SKU 88-210, or someone's last name, vector search often returns results that are topically related but not the target entity. Keyword search like BM25 handles these cases perfectly.

Run both retrievals simultaneously and merge the result lists. Reciprocal Rank Fusion (RRF) is a simple and robust way to merge, because it only uses the rank within each list and doesn't depend on incomparable scores between the two systems:

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

k=60 is the value from the original RRF paper and is a reasonable default.

BM25 is a classic ranking function that has been used in information retrieval for decades, belonging to the sparse retrieval family. It represents both documents and queries as sets of terms, with a core idea based on two points: the higher the term frequency (TF), the more important the term is to the document; but the more common the term is in the corpus (the lower the inverse document frequency, IDF), the less discriminative it becomes. BM25 also introduces document length normalization on top of this, preventing long documents from dominating simply because they contain more words. It does no semantic generalization at all, so it is extremely sensitive and precise for rare strings like E1042 and 88-210, but it cannot handle synonymous paraphrasing—which is exactly why it complements dense vector retrieval.

RRF (Reciprocal Rank Fusion) solves the problem of how to merge results from multiple retrieval paths. Similarity scores from different retrieval systems are often not on the same scale, so directly weighting and summing them is unreliable. RRF simply discards the scores and looks only at each document's rank in each result list, accumulating a score with 1/(k+rank)—the higher the rank, the greater the contribution. The constant k acts as a smoothing factor, preventing the top-ranked document from having too much weight; k=60 comes from the original 2009 paper by Cormack et al., and in practice anything from 20 to 100 performs stably. Because it relies only on ranks, RRF is naturally robust across different retrievers and requires almost no tuning.

Rerank the Candidates

Embedding search compares the query vector against precomputed chunk vectors. Cross-encoder rerankers, on the other hand, read the query and each text chunk together, making more accurate relevance judgments—but they are too slow to run over the full corpus. The practical approach is: retrieve 50 candidates with a low-cost method, then rerank and keep the top 5.

A cross-encoder is architecturally fundamentally different from the bi-encoder that generates embeddings. A bi-encoder encodes the query and document separately into two independent vectors and compares them using cosine similarity; document vectors can be precomputed offline, enabling fast retrieval over the full corpus. A cross-encoder instead concatenates the query and document into a single input fed into a Transformer, allowing the attention mechanism to interact directly between the two, capturing finer-grained semantic relationships and producing noticeably more accurate relevance judgments. The cost is that it cannot be precomputed: every (query, document) pair requires one forward pass, which is completely infeasible for a corpus of millions. Therefore the standard practice is to use a cross-encoder only for fine-ranking the dozens of candidates returned by retrieval, forming a two-stage pipeline of "coarse recall + fine ranking." Common open-source models include the BERT- or MiniLM-based cross-encoder series; smaller and faster versions are better suited to latency-sensitive scenarios.

Filter First, Then Rank

If a question is limited to documents for a specific product, year, or customer, apply it as a metadata filter rather than hoping similarity will distinguish automatically. Permission filtering is especially critical: filter out documents the user has no right to see before retrieval. Writing "do not mention confidential documents" in the prompt does not constitute access control.

Metadata filtering refers to using structured fields (such as product name, year, customer ID, document type, access permission tags) to pre-filter the candidate set at the retrieval stage, rather than letting similarity scores bear the burden of discrimination themselves. There are two common implementations: pre-filtering first narrows the document set by conditions and then performs vector search, ensuring results don't go out of bounds, though it sacrifices some index efficiency; post-filtering retrieves first and then removes non-matching items, which is fast but may lead to insufficient valid results. Mature vector databases (such as FAISS wrappers with metadata fields, Qdrant, Weaviate, and pgvector combined with SQL conditions) usually support pre-filtering. Permission filtering is the highest-security category of metadata filtering: documents the user has no right to access must be excluded before retrieval based on the user's access control list. Any approach that relies on the model "voluntarily not mentioning" prompt constraints does not constitute real access control—once a user uses a leading question, confidential content may leak.

Prepare for "Not Found"

Some questions genuinely have no answer in the data. If the highest score after reranking falls below a threshold validated against the test set, tell the model that no relevant content was found and let it respond honestly. "I can't find this information in the documents" is a good answer. A confident answer stitched together from loosely related text chunks is the worst possible outcome.

One-line Summary

Most RAG wrong answers stem from retrieval failure; first quantify retrieval with recall@k and MRR, then optimize chunking, hybrid search, reranking, and filtering one by one.

Paragraph Summary

The article points out that when a RAG system gives a wrong answer, the problem is usually not the generation model but the retrieval stage—the correct text chunk was never recalled, so even a strong model has nothing to work with. Therefore, one should first step away from the generation model and build a retrieval evaluation baseline using 50 to 100 annotated real questions, quantifying retrieval quality with two metrics: recall@k and MRR. On this basis, optimize in sequence: split by document structure (headings, paragraphs, table rows) and attach heading path prefixes; combine BM25 keyword retrieval with vector retrieval and fuse results via RRF to balance exact strings and semantic matching; rerank a small set of candidates with a cross-encoder; move metadata and permission filtering ahead of retrieval; and set a score threshold for questions with genuinely no answer, letting the model honestly state that it cannot find the information.