ReRankEval
An open-source hybrid RAG retrieval evaluation pipeline that integrates dense vector retrieval, BM25 keyword retrieval, Reciprocal Rank Fusion (RRF), and LLM-based reranking. It supports side-by-side comparison evaluation of multiple retrieval strategies.
Timeline (last 90 days)
Developer Saurabh Kamal open-sourced a hybrid retrieval pipeline called ReRankEval, along with a complete codebase and video walkthrough.
The ReRankEval project uses a prompt-driven LLM reranker, allowing the large model to directly judge the relevance of each candidate document to the question and reorder them.
The project introduces three standard retrieval evaluation metrics: Hit Rate, MRR (Mean Reciprocal Rank), and NDCG (Normalized Discounted Cumulative Gain).
The author used five real financial/payment documents and ten manually verified test questions to conduct a side-by-side comparison of four approaches: vector-only, BM25-only, hybrid retrieval without reranking, and the full ReRankEval pipeline.
The project adopts a layered directory structure of ingest → vector_store → sparse_retriever → fusion → reranker → pipeline → generate → eval, decoupling the ingestion side from the query side.
LLM reranking has high computational overhead, but when applied to only a small number of candidates the overall cost is manageable, and the precision improvement is often significant.
All Facts (6)
The ReRankEval project uses a prompt-driven LLM reranker, allowing the large model to directly judge the relevance of each candidate document to the question and reorder them.
50%UnverifiedThe project introduces three standard retrieval evaluation metrics: Hit Rate, MRR (Mean Reciprocal Rank), and NDCG (Normalized Discounted Cumulative Gain).
50%UnverifiedThe author used five real financial/payment documents and ten manually verified test questions to conduct a side-by-side comparison of four approaches: vector-only, BM25-only, hybrid retrieval without reranking, and the full ReRankEval pipeline.
50%UnverifiedThe project adopts a layered directory structure of ingest → vector_store → sparse_retriever → fusion → reranker → pipeline → generate → eval, decoupling the ingestion side from the query side.
50%UnverifiedLLM reranking has high computational overhead, but when applied to only a small number of candidates the overall cost is manageable, and the precision improvement is often significant.
50%UnverifiedDeveloper Saurabh Kamal open-sourced a hybrid retrieval pipeline called ReRankEval, along with a complete codebase and video walkthrough.
50%