The RAG Evaluation Dilemma: If a Multi-Document Query Only Recalls One, Does It Count as a Hit?

An Overlooked Blind Spot in Evaluation

Suppose a user's question requires two documents at once: one is the current policy document, and the other is a separate exception clause explanation. The retriever only returned the policy document and missed the exception clause.

How should your evaluation metric record this case?

Under the rule of "any expected document in top-k," this counts as a hit. Under the rule of "all required evidence in top-k," this is an incomplete result. The two metrics give completely opposite conclusions, and they are not interchangeable.

This is a case recently discussed in the Reddit community. Although it is only an illustrative scenario rather than production data, it precisely hits a long-obscured issue in RAG evaluation: in multi-document query scenarios, what is the definition of a "hit"?

If you are building a RAG system with LangChain or a similar framework, and using metrics like hit rate and recall@k to compare the effectiveness of different retrieval approaches, then if this question is not clarified, your selection decision may be wrong from the very beginning.

Why the Definition of a "Hit" Splits

The default logic of most RAG evaluation tools is "any-hit": as long as any document labeled as relevant appears in the Top-K results, it counts as a hit.

This design is fine in single-document question-answering scenarios—one question corresponds to one piece of evidence, and a hit is a hit. But in reality, a large number of queries are composite: comparative analysis requires two documents, causal reasoning requires two related sources before and after, and regulatory compliance questions require the main clause plus exception explanations.

When a question inherently requires multiple pieces of evidence, any-hit records "getting only half" and "getting the complete set" as the same result. This directly leads to two serious consequences:

Inflated metrics. A retrieval approach may appear to have high recall, but in reality many queries only recall part of the required evidence. Users receive incomplete information, and the model may generate incorrect answers based on one-sided materials.

Distorted method comparison. Two retrieval strategies may score similarly under the any-hit metric, but differ enormously under the complete evidence metric. You think you are choosing the better approach, but in fact you are only choosing two variants of the same approach.

The difference between hit rate and recall@k: hit rate usually refers to "the proportion of queries that hit at least one relevant document," a binary judgment that does not care about the rank of the hit; recall@k measures what proportion of all relevant documents are recalled in the Top-K results, and is a continuous value. In single-document scenarios, the two trends are consistent, but in multi-document queries, hit rate naturally leans toward any-hit logic, while recall@k, if calculated as "number of relevant documents recalled / total number of relevant documents," is closer to the all-evidence perspective. This is also why the same retriever may perform very differently under different metrics—the judgment rules behind the metrics differ, and so do the capability dimensions they reflect. When selecting a method, you should clarify whether what you want to optimize is "can it find it" or "can it find it all."

Before Comparing Retrieval Approaches, Align These Four Things

The poster proposed a very practical checklist. Before using two retrieval scores for comparison, confirm each item one by one:

• Do the two evaluators use the same success judgment rule? If one uses any-hit and the other uses all-evidence, the scores are not comparable.
• Are acceptable alternative documents handled consistently? Some queries have multiple equivalent correct answers. If evaluators differ in their recognition of "alternative documents," the results are likewise incomparable.
• Are the expected objects documents or chunks? Is the ID mapping consistent? For the same knowledge base content, some label by whole document and some label by chunk. Different granularity leads to completely different hit judgments.
• Are missing annotations explicitly recorded rather than silently treated as irrelevant? This is the most hidden pitfall. If some queries in the evaluation dataset have no complete relevance annotations at all, any-hit will treat "unannotated" as "irrelevant," thereby underestimating actual recall capability.

These four checks look basic, but they explain why many teams repeatedly argue about RAG evaluation—on the surface everyone is debating an average score, but in fact even the scoring rules have not been unified.

Document granularity vs chunk granularity: Document-level evaluation treats the whole document as the unit of hit; as long as the retrieval result falls within that document, it counts as a hit. Chunk-level evaluation requires the specific retrieved passage to match the labeled relevant passage. The choice of granularity directly affects metric values: document-level is looser and tends to produce high scores but may conceal retrieval precision problems; chunk-level is stricter and closer to the context actually used in the generation stage, but places higher demands on annotation quality. A common problem in practice is that the evaluator uses chunk-level retrieval but document-level annotation, causing ID mapping misalignment and systematic bias in hit judgment. Unifying granularity and establishing a stable doc-to-chunk ID mapping are prerequisites for reproducible evaluation.

The Truly Useful Output: Not a Score, but a Difference List

The poster gave a very valuable suggestion: rather than arguing about which metric has a higher average score, it is better to output a list of affected queries.

Specifically, this includes:

1. Which queries produced different judgments under the two rules
2. What the expected evidence was for each query
3. The actual retrieved position (rank)
4. The specific rule that caused the conclusion to flip

The value of this list is that it turns "metric disagreement" from a subjective argument into a reproducible fact. Team members can inspect item by item: so it was this query's second document that was not recalled; any-hit recorded a hit, all-evidence recorded a miss. The focus of discussion shifts from "which metric is better" to "does this query actually need two documents," and the latter is the question that has an answer.

any-hit, all-evidence, or both?

Back to the poster's final question: in actual deployment decisions, which would you choose?

From an engineering practice perspective, the two metrics answer different questions:

any-hit measures "whether the retriever has the ability to find relevant information." It is suitable for troubleshooting whether the retriever has completely failed—if it cannot even find any relevant document, there is a serious problem in the retrieval stage.

all-evidence measures "whether the retriever can support a complete answer." It is directly related to final generation quality—incomplete evidence leads to model hallucination or incomplete answers.

In deployment decisions, all-evidence should be the primary reference metric, because it is closer to the actual user experience. But any-hit can be retained as a diagnostic tool: when the all-evidence score is low, checking whether any-hit is normal can quickly distinguish whether it is "found nothing at all" or "found something but not everything."

The gap between these two metrics is itself a valuable signal. If any-hit is high while all-evidence is low, the problem lies in coverage—you may need to increase Top-K, improve the chunk strategy, or introduce multi-hop retrieval. If both are low, then the retriever's basic capability is problematic.

Multi-hop retrieval: When a question requires chaining multiple documents to answer, a single retrieval often cannot recall all the evidence at once. In this case, multi-hop retrieval is needed—first retrieve the first batch of documents, construct a new query based on their results, then retrieve the next batch, gradually assembling the complete evidence chain. It naturally fits the all-evidence metric: low all-evidence scores with normal any-hit are precisely a typical signal for applying multi-hop retrieval. In implementation, it can be combined with query rewriting, iterative retrieval, or graph structure traversal. Note that multi-hop introduces additional latency and error accumulation, and whether it is worth it depends on the business requirement for evidence completeness.

There Is No Universal Answer for Multi-Document Query Evaluation

It must be acknowledged that evaluating multi-document queries is itself much more complex than single-document evaluation. What kind of document combination counts as "complete evidence" often depends on the specific task and business scenario. The standard for "complete" in a legal question-answering system and the standard for "complete" in a technical documentation assistant may be completely different.

But at least one thing can be done: explicitly declare which rule is used in the evaluation process, and ensure that both sides being compared use the same rule. Turning implicit assumptions into explicit conventions is the first step in moving RAG evaluation from "arguing" to "engineering."

One-line Summary

In multi-document queries, any-hit and all-evidence judgments are opposite; deployment should prioritize all-evidence and explicitly declare the evaluation rule.

Paragraph Summary

The article points out an overlooked blind spot in RAG evaluation: when a query requires multiple documents, any-hit (a hit if any one is retrieved) and all-evidence (all required evidence must be included) can give completely opposite judgments, and they are not interchangeable. any-hit leads to inflated metrics and distorted method comparison. The author suggests aligning four rules before comparing retrieval approaches: success judgment criteria, handling of alternative documents, document vs chunk granularity and ID mapping, and handling of missing annotations. More useful than an average score is outputting a list of affected queries to make disagreements reproducible. In deployment, all-evidence should be the primary metric, with any-hit as a diagnostic tool; the gap between them can indicate whether the issue is coverage or basic retrieval capability.