RAG Reranking in Practice: A Four-Way Reranker Comparison and Selection Guide

The performance bottleneck of Retrieval-Augmented Generation (RAG) systems often lies not on the generation side, but on the retrieval side. When vector search returns Top 20 results mixed with a large number of text chunks that are "topically relevant but cannot answer the question," an appropriate Reranker becomes the key link that determines final answer quality. Developer Saurabh Kamal published a complete RAG reranking comparison project on Reddit and YouTube, testing four reranking approaches on 5 payment-industry PDFs (covering payment regulations, PCI DSS, and central bank reports, totaling 1507 text chunks).

Why Reranking Is Needed: Similarity Is Not Relevance

The working principle of vector search determines its capability boundary. After mapping questions and text chunks into a high-dimensional space through an embedding model, cosine similarity measures the closeness of semantic direction—it is good at finding passages that "discuss the same topic," but cannot determine whether a passage truly answers the question.

The experiments in the project confirm this: a text chunk can be adjacent to the query in vector space while containing none of the required information. This is exactly why RAG pipelines need a two-stage retrieval architecture—the first stage uses vector search to quickly narrow the scope (recalling Top 20), and the second stage uses a reranker to finely judge the true relevance of each text chunk, ultimately sending only the Top 5 to the LLM. Reducing the number of text chunks sent to the LLM directly lowers token consumption and inference latency, while also reducing the interference of noise on generation quality.

The Essential Difference Between Bi-Encoder and Cross-Encoder

Understanding the technical principles of reranking requires first distinguishing two types of encoding architectures. A Bi-Encoder (dual-tower model) independently encodes the query and document into vectors, then calculates distance through cosine similarity or dot product. The advantage of this architecture is that document vectors can be precomputed and indexed; during retrieval, only the query needs to be encoded once to complete approximate nearest neighbor search over millions of vectors, with extremely low latency. But the cost is that the query and document have no interaction during the encoding stage, so the model can only capture their respective global semantics and easily misses the fine-grained signals needed for question-answer matching.

A Cross-Encoder concatenates the query and document into a single sequence and feeds it into a Transformer, allowing the self-attention mechanism to directly model the association between query terms and document terms at every layer. This cross-encoding can capture structural information such as "whether the key entity in the question appears in the answer position of the document," so ranking accuracy is usually significantly better than that of a Bi-Encoder. But the cost is that every scoring operation must rerun the full forward inference and cannot be precomputed. It is acceptable at the Top 20 scale, but infeasible on million-level corpora. This is the fundamental motivation for the two-stage retrieval architecture: use a Bi-Encoder for coarse filtering and a Cross-Encoder for fine ranking.

Technical Route Comparison of Four Reranking Approaches

The project implements a unified pipeline: question → retrieve Top 20 → rerank → Top 5 → LLM → answer, and through a shared auxiliary interface allows the four rerankers to be swapped in interchangeably.

Local Cross-Encoder: Fast but Not Necessarily Accurate

The Cross-Encoder is fundamentally different from the Bi-Encoder used in vector search. The Bi-Encoder separately encodes the question and text chunk, then compares vector distance; the Cross-Encoder concatenates the question and text chunk and feeds them into the model together, allowing the attention mechanism to directly capture fine-grained interactions between the two. The project uses cross-encoder/ms-marco-MiniLM-L-6-v2, a lightweight model that runs locally based on Hugging Face Transformers and PyTorch.

But the experimental results were surprising—this small model actually made the ranking worse. Its behavior pattern leans toward lexical matching, is especially sensitive to the shuffled text sequence tables in PDFs, and ultimately pushed two correct answer text chunks out of the Top 5. This reminds us: a reranker is not necessarily better the larger it is, nor safer the smaller it is; the key is whether it fits your data characteristics.

Technical Background of MS MARCO and the MiniLM Model

The cross-encoder/ms-marco-MiniLM-L-6-v2 used in the project comes from the ranking model series trained on Microsoft's MS MARCO dataset. MS MARCO (Microsoft MAchine Reading COmprehension) is a large-scale English question answering dataset containing more than 500,000 real search queries and their relevant passage annotations, and has long served as a benchmark in the information retrieval field. This series of models is based on the MiniLM architecture—a lightweight Transformer compressed from large models such as BERT through knowledge distillation, where L-6 indicates 6 layers and v2 is the second version.

The MiniLM-type Cross-Encoder is designed for general web search scenarios, and its training data mainly consists of natural language passages. When transferred to text parsed from PDFs, it encounters two domain shift problems: first, tables, lists, headers, and footers in PDFs are parsed into shuffled text sequence tables, destroying the continuous semantic structure the model relies on; second, payment regulation documents heavily use clause numbers (such as "3.3" and "Requirement 3"), and the tokenizer may split these symbols into fragments, causing lexical matching signals to become disordered. This explains why the model performs well on general benchmarks but shows ranking degradation on industry-specific data.

LLM Reranking: Strongest Comprehension but High Cost

The LLM-based Listwise reranking strategy lets a large model directly read all candidate text chunks and rank them. In this project, the LLM reranker demonstrated deep understanding of question structure—for example, it correctly judged that "3.3" belongs to "Requirement 3" and ranked the two text chunks that truly contained the answer at 1st and 2nd place respectively. This grasp of contextual logical relationships is difficult for a Cross-Encoder to match.

API Reranking: The Engineered Choice of Cohere and Jina

The Cohere Rerank API and Jina Reranker API represent another approach—outsourcing reranking to specially optimized commercial services. In the experiments, Cohere promoted an answer text chunk originally ranked 15th to 1st place, turning a correct answer that was about to be missed into a valid response with cited sources.

According to the project author's analysis in the YouTube video, Cohere and Jina achieved the best balance between accuracy and speed on this payment-industry dataset. But the API approach also brings two trade-offs that need to be weighed: costs billed by number of calls, and privacy considerations involved in sending document content to a third party.

Billing Models and Service Architecture of Reranking APIs

Cohere Rerank and Jina Reranker are managed reranking services. Developers submit queries and candidate document lists through HTTP APIs, and the server returns ranking results with relevance scores. Such services are usually billed per "1,000 search units," where one search unit is defined as one query paired with up to 100 documents. Taking Cohere Rerank 3 as an example, pricing is about $2 per 1,000 searches. In a Top 20 scenario, each query consumes 0.2 search units, so the cost is relatively controllable.

From a service architecture perspective, these APIs also run Cross-Encoder-type models underneath, but they have undergone multilingual and multi-domain fine-tuning for commercial scenarios and are deployed on dedicated inference clusters to guarantee throughput and latency. On privacy, note that submitted queries and document content leave the local environment. For sensitive data such as payment regulations and medical records, compliance risks need to be assessed. Some services provide data non-retention commitments or on-premises deployment versions, but prices increase accordingly. Engineering-wise, network jitter, rate limits, and fallback strategies also need to be handled—additional complexity that local models do not require.

Evaluation Methodology: How to Scientifically Compare Rerankers

This project has a principle in evaluation design worth borrowing: all rerankers must run on exactly the same Top 20 results. If different approaches use different recall sets, ranking differences cannot be attributed to the reranker itself.

The evaluation metrics include three items: position changes before and after reranking (intuitively showing how much each text chunk was promoted or demoted), Hit@5 (whether the correct answer appears in the final Top 5), and average latency. The code structure is clearly split into four modules: retriever.py, rerankers.py, generate.py, and compare.py, making reproduction and secondary development convenient.

On the tech stack, the project uses Qdrant as the vector database (cosine similarity), gemini-embedding-2-preview to generate 3072-dimensional embedding vectors, combined with Hugging Face Transformers, PyTorch, and the three reranking approaches mentioned above.

Practical Meaning of Hit@K and MRR Metrics

Hit@5 measures whether the correct answer appears in the final top 5 results. If it appears, it scores 1; otherwise, it scores 0, and the average over multiple queries gives the hit rate. This metric directly corresponds to the actual needs of a RAG system—only text chunks that enter the Top 5 are sent to the LLM as context. Compared with more academic NDCG or MAP, Hit@K is closer to engineering decisions because it does not care whether the correct result ranks 1st or 5th, only whether it is truncated.

When a query has multiple correct answers, it can be extended to Recall@5, that is, what proportion of correct text chunks are covered in the Top 5. The case in the project with two correct answers is well suited to this metric: LLM reranking ranked them 1st and 2nd respectively, so Recall@5 is 2/2; while the MiniLM Cross-Encoder pushed both out of the Top 5, so Recall@5 is 0. Average latency should be measured under the same hardware and batch size. For API approaches, network round-trip time should also be distinguished from server-side inference time to avoid mistaking network overhead for model performance differences.

Selection Advice: There Is No Universal Solution, Only a Fitting One

Based on the experimental data and experience from this project, reranking solution selection can follow several principles. A local Cross-Encoder is suitable for scenarios with extremely high latency and privacy requirements, but it is necessary to first verify with actual data whether it causes ranking degradation—the performance of MiniLM in this project is a negative example. LLM reranking is suitable for scenarios with strict accuracy requirements and willingness to bear higher inference costs. API reranking provides a good balance among accuracy, speed, and engineering complexity, but requires accepting constraints around call costs and data egress.

The most core lesson comes from the project author's conclusion: always measure, never guess. Run all candidate rerankers on the same recall set, and let the data tell you which one best fits your business scenario.

The complete code and video explanation are available in the GitHub repository (saurabhkamal/RAG-ReRanker-Comparison) and on the YouTube channel.

One-line Summary

Four RAG reranking approaches were tested on 5 payment-industry PDFs, revealing that similarity is not relevance.

Paragraph Summary

The project compared four reranking approaches—local Cross-Encoder, LLM Listwise, Cohere API, and Jina API—on 1507 payment-industry text chunks. The core finding is that cosine similarity in vector search can only find topically relevant passages and cannot determine whether they truly answer the question, so a two-stage retrieval architecture is needed. In the experiments, the lightweight MiniLM Cross-Encoder, biased toward lexical matching, pushed correct answers out of the Top 5; LLM reranking had the strongest comprehension but high cost; Cohere and Jina APIs achieved the best balance between accuracy and speed. The evaluation methodology emphasizes that all rerankers must run on the same Top 20 recall set and focuses on three metrics: position changes, Hit@5, and latency. The author's core advice is: always measure, never guess.