Agentic RAG Benchmark: Agent Loop Crushes Traditional Pipelines with 92.7% Accuracy

A Benchmark Result That Surprised RAG Practitioners

The team behind open-source project PipesHub ran a controlled experiment on Google's FRAMES benchmark, and the results are striking: a carefully tuned traditional RAG pipeline topped out at 78.9% accuracy, while a simple Agent loop — one that can read retrieval results and search again — scored 92.7%. That figure is nearly equivalent to directly feeding the model the correct documents.

FRAMES is a dataset specifically designed to test multi-hop reasoning, and this experiment covered all 824 multi-hop questions. To ensure fairness, all pipeline variants used the same model, the same embeddings, and the same document corpus — the only variables were retrieval and orchestration strategies. The team built 18 pipeline variants in total to isolate the true contribution of each component.

FRAMES (Factuality, Retrieval, And Multi-hop Evaluation of Systems) is a multi-hop QA benchmark dataset released by Google DeepMind in 2024, specifically designed to evaluate how well systems perform when they need to synthesize information across multiple documents. Unlike single-hop QA, multi-hop reasoning requires a system to first retrieve an intermediate fact, then use that as a clue to continue searching, ultimately assembling a complete answer. For example, answering "which city is the university of the scientist who won Award X located in?" requires first finding the award winner's name, then looking up their institution. FRAMES therefore exposes structural weaknesses in retrieval strategies far more effectively than traditional QA benchmarks, and is considered a rigorous test of a RAG system's true upper bound.

Do Hybrid Search, Reranking, Query Decomposition, and Query Expansion Actually Help?

In mainstream RAG practice, hybrid search, reranking, query decomposition, and query expansion are treated almost as "standard equipment." But the core question this experiment set out to answer was: how much does each of these components actually contribute?

The most counterintuitive finding involved the reranker. The team found that a small reranker actually dropped the best pipeline's accuracy by 9 percentage points, and switching to a larger reranker produced almost no improvement either. The authors admitted they had previously suspected that reranking offered limited benefit on this task — and the experiment confirmed that hypothesis.

This is a wake-up call for engineering teams that default to the assumption that "adding a reranker always helps": a component's value is highly dependent on task type and data distribution. Multi-hop QA scenarios don't necessarily benefit from generic retrieval-augmentation techniques. Blindly stacking components can be not just ineffective, but actively harmful.

Hybrid search combines sparse retrieval (e.g., BM25 keyword matching based on term frequency) with dense retrieval (semantic similarity search based on embeddings), typically using fusion algorithms like RRF (Reciprocal Rank Fusion) to merge results from both approaches. The goal is to capture both precise keyword matches and semantic generalization. A reranker is a second-stage ranking model introduced after initial retrieval, commonly implemented as a cross-encoder that scores query-document pairs more precisely to filter the most relevant documents from the top-K candidates. Both techniques tend to deliver consistent gains in single-hop QA and keyword retrieval scenarios — but in multi-hop settings, the bottleneck isn't single-pass recall quality; it's whether the system can dynamically adjust its search direction. The marginal value of these "refinement" components can therefore be far lower than expected.

Why the Agent Loop Pulls Ahead

Traditional RAG is fundamentally a linear "retrieve once, generate once" pipeline, where retrieval quality defines the answer ceiling. The Agent loop introduces iterative capability: the model can examine the first round of results, judge whether the information is sufficient, and if not, issue another search.

In multi-hop questions, answers are often scattered across multiple documents, and you need to find clue A before you even know to look for clue B. Linear pipelines struggle to handle this kind of dependency chain, while the Agent loop is a natural fit — it transforms "retrieval" from a one-shot action into a tool that can be called repeatedly.

The 92.7% score approaches the theoretical ceiling of "providing the correct documents directly to the model," which suggests that with the right orchestration, retrieval gaps can be compensated for through multiple rounds of self-correction. This also echoes the broader industry trend of migrating from static RAG toward Agentic RAG.

An Evaluation Pitfall That's Easy to Overlook

The team also observed a phenomenon that all RAG evaluators should be wary of: even when explicitly instructed to answer only based on retrieved documents, models will still fill in gaps from memory — and these "hallucinated" answers often come with citations that look perfectly legitimate.

In other words, having citations does not mean having grounding. If you only check whether the final answer is correct and whether it has a citation, you'll likely count answers that the model got right from parametric memory as retrieval successes, systematically overestimating the system's actual capabilities.

To address this, the team performed a secondary verification on every "correct answer" — cross-checking whether the content the system actually read matched the answer it produced. This rigorous faithfulness check is the key step that makes benchmark numbers truly trustworthy, and it's one that's frequently skipped in RAG evaluations.

This phenomenon is known as "hallucinated citations" or parametric knowledge leakage. Large language models memorize vast amounts of factual knowledge during pretraining, and even in a RAG setup where they're instructed to answer based on context, they will still blend parametric memory with retrieved content when retrieval is insufficient — and automatically generate properly formatted source attributions, making them difficult to distinguish from genuinely document-grounded answers. This means "answer accuracy" and "retrieval faithfulness" are two independent evaluation dimensions. Faithfulness evaluation requires verifying that every key claim in an answer can be directly traced back to the retrieved documents, not merely checking whether the conclusion is correct. Many current RAG evaluation frameworks (such as RAGAS) already list faithfulness as a core metric, but in practice this step is often simplified or skipped entirely.

Implications for Engineering Practice

This experiment offers several directly actionable takeaways. First, don't blindly trust "standard components" — every optimization should be A/B tested on your own data, and rerankers in particular deserve careful evaluation. Second, for complex queries involving multi-hop reasoning or information synthesis, an Agent loop's iterative retrieval may deliver better returns than stacking single-pass pipeline techniques. Third, during evaluation, you must verify whether answers actually come from retrieved content — otherwise accuracy metrics will be distorted.

It's worth noting that the authors work on the open-source project PipesHub, and the benchmark code and runbooks have been published in the repository. This is both a disclosure of potential bias and a sign that the findings can be independently reproduced — which is refreshingly rare in today's RAG discourse, which is often saturated with marketing language.

That said, the conclusions should be taken with appropriate perspective: they are based on a single dataset, FRAMES, focused specifically on multi-hop QA, and whether they generalize to other task types still requires further validation. But the experiment at least provides a clear signal — the Agentic paradigm has a structural advantage in complex retrieval scenarios.