Hybrid RAG Retrieval in Practice: A Complete Guide to Dense+BM25+RRF and Reranking

The effectiveness of Retrieval-Augmented Generation (RAG) depends heavily on whether the retrieval stage can surface truly relevant documents for the LLM. Developer Saurabh Kamal recently open-sourced a hybrid retrieval pipeline called ReRankEval, complete with full code and a video walkthrough. The system chains together dense vector retrieval, BM25 keyword retrieval, Reciprocal Rank Fusion (RRF), and LLM-based reranking into a complete pipeline — then rigorously benchmarks it against real financial/payment documents.

This article draws from his Reddit post, GitHub repository, and YouTube tutorial to break down the core design decisions and what they mean for building production-grade RAG systems.

Why Hybrid Retrieval?

Every single-strategy retrieval approach has its blind spots. Dense vector search excels at capturing semantic similarity — it can connect "installment plan" with "installment" — but tends to stumble on exact matches for proper nouns, code snippets, and numbers. Sparse keyword retrieval like BM25 is the opposite: highly sensitive to exact term matching, but unable to understand paraphrasing or contextual semantics.

The logic behind hybrid retrieval is straightforward complementarity: dense retrieval handles semantic recall, BM25 handles keyword recall, and the two result sets get merged. The author is explicit about this — the project isn't about assuming one approach is better, but about fairly comparing multiple strategies on the same test set and letting the numbers speak.

RRF: Solving the Score Incompatibility Problem

Merging two retrieval result sets runs into a thorny problem: vector search returns cosine similarity or distance scores, while BM25 returns term-frequency-based scores. The two scales are completely different — you simply can't add or compare raw scores directly.

Reciproal Rank Fusion (RRF) was designed exactly for this. Instead of looking at raw scores, it only cares about each document's rank position in each result list. The core formula is:

RRF_score(d) = Σ 1 / (k + rank_i(d))

Here, rank_i(d) is document d's rank in the i-th retrieval result, and k is a smoothing constant (commonly set to 60). Higher-ranked documents contribute more to the score; documents that rank highly across multiple retrieval paths accumulate higher combined scores and rise to the top after fusion. This "rank-only, score-blind" approach elegantly sidesteps the normalization problem with heterogeneous scores.

The Fundamental Difference Between Rerankers and Retrievers

Many people mistake reranking for just another round of retrieval, but the author emphasizes a fundamental architectural difference.

A retriever must rapidly filter a candidate set from thousands or millions of documents, prioritizing speed and recall. It typically uses a bi-encoder architecture that encodes queries and documents independently. A reranker, by contrast, only processes the small candidate set returned by the retriever (say, Top-K). It takes a query and each candidate document as a pair, jointly assessing relevance — the classic cross-encoder approach.

This project uses a prompt-driven LLM reranker, asking the language model to directly judge each candidate's relevance to the question and reorder accordingly. While computationally expensive per inference, it only operates on a small candidate set, keeping overall cost manageable — while delivering substantial precision gains.

---

The architectural difference between bi-encoders and cross-encoders determines where each fits best. Bi-encoder models encode queries and documents independently into vectors; the query vector is computed once, and document vectors can be precomputed offline and stored in an index, enabling millisecond-scale approximate nearest neighbor (ANN) search over millions of documents. The trade-off is that all interaction information between query and document is compressed into two independent vectors, limiting fine-grained semantic matching. Cross-encoders concatenate the query and candidate document and pass them through a Transformer together; the self-attention mechanism can capture fine-grained token-level interactions between the two, yielding significantly higher relevance accuracy — but every inference requires a fresh forward pass with no precomputation possible, and cost scales linearly with the number of candidates. This is why the industry widely adopts a two-stage "bi-encoder coarse recall + cross-encoder fine ranking" architecture. ReRankEval substitutes a dedicated cross-encoder with an LLM for reranking — the same underlying principle.

---

How to Rigorously Evaluate a RAG Pipeline

Perhaps the most valuable aspect of this project is that it doesn't stop at "feels more accurate" — it introduces three standard retrieval evaluation metrics:

• Hit Rate: Whether the correct document appears anywhere in the returned results, measuring baseline recall capability.
• MRR (Mean Reciprocal Rank): The higher the correct answer ranks, the higher the score. Measures where the first correct result appears.
• NDCG (Normalized Discounted Cumulative Gain): Accounts for the ranking quality of multiple relevant documents — the most comprehensive ordering metric.

The author uses five real financial/payment documents and ten manually verified test questions to compare four strategies head-to-head: vector-only, BM25-only, hybrid without reranking, and the full ReRankEval pipeline. It wraps up with a comparison table of real numbers, making each strategy's strengths and weaknesses immediately clear.

---

The calculation logic behind MRR and NDCG is worth a brief elaboration. MRR takes the reciprocal of the rank of the first correct document for each query, then averages across all queries: a correct document at rank 1 scores 1.0, rank 2 scores 0.5, rank 3 scores 0.33, and so on. It's most sensitive to scenarios where "finding at least one answer, and finding it quickly" matters most. NDCG introduces a "discount" concept — relevant documents ranked lower contribute to the score with logarithmic decay, normalized against the ideal ranking's DCG. This lets it distinguish between "2 relevant documents in the Top-3" versus "2 relevant documents in the Top-10" — making it better suited for measuring overall ranking quality when returning multiple candidates. In a RAG context, a system might have high Hit Rate (correct documents always appear in Top-10) but low MRR (correct documents frequently land at positions 8 or 9). Using both metrics together exposes the true bottleneck in retrieval ranking.

---

Breaking Down the Production-Grade Code Structure

Beyond the algorithms, the project provides a clean, modular engineering structure that splits the RAG pipeline into clearly responsible components:

ingest → vector_store → sparse_retriever → fusion → reranker → pipeline → generate → eval

This layered approach fully decouples the ingestion side (document parsing, chunking, index building) from the query side (retrieval, fusion, reranking, generation), making it easy to test each stage in isolation and swap out components.

On the tech stack: the project uses Python with Qdrant as the vector database, rank_bm25 (BM25Okapi) for sparse retrieval, EURI LLM Gateway for chat and embedding models, pdfplumber for page-level PDF text extraction, and custom implementations for fusion and reranking. The full combination covers the entire pipeline from document parsing to answer generation.

Takeaways for RAG Developers

This open-source project's value isn't in any single novel technique — it's in the way it integrates the complete engineering pipeline and evaluation methodology for hybrid retrieval into one place. For developers building RAG applications, a few key points are worth keeping in mind: don't over-rely on a single retrieval strategy; use rank-fusion methods like RRF to sidestep score incompatibility; treat reranking as the "last mile" for precision; and most importantly, use quantitative metrics like Hit Rate, MRR, and NDCG for fair comparisons rather than relying on intuition.

The full code is open-sourced on GitHub — interested developers can clone it, reproduce the results, and adapt it to their own datasets. The author has also labeled this as Part 1, so more in-depth content may follow.