How Annotation Coverage Distorts RAG Retrieval Scores: An Empirical Study on Breast Lesions

When the "Standard Answer" Expands, Why Do Retrieval Scores Stay Frozen?

In evaluating clinical RAG systems, we habitually use retrieval scores (Recall@K, MRR, nDCG) to judge system quality. But an empirical study on breast lesion management literature reveals an easily overlooked problem: retrieval scores are highly dependent on annotation coverage. When researchers expanded evidence snippets from 25 to 37 and acceptable chunk targets from 36 to 51, the retrieval rankings did not change at all. This means the scores we saw previously may have been artificially inflated or deflated under incomplete annotation.

The study constructed a small corpus of 18 documents and a development benchmark of 18 queries, each corresponding to a yes/no or WH question within one document. The retrieval pipeline fused dense vectors, BM25, and cross-encoder reranking, and systematically compared chunk sizes from 128 to 1024 tokens, overlap rates from 0% to 20%, and four retriever configurations. Relevance was no longer defined at the document level but required evidence snippets to map to all chunks containing them—overlapping chunks containing equivalent evidence were all considered relevant.

RAG (Retrieval-Augmented Generation) is a technical architecture that combines information retrieval with large language model generation: the system first retrieves document snippets relevant to the user query from an external knowledge base, then feeds these snippets as context to the generative model, which produces an answer based on the retrieved evidence. In clinical settings, the value of RAG lies in making model answers "grounded," reducing hallucination risk. The quality of the retrieval stage directly determines the ceiling of generated content, so retrieval metrics like Recall@K, MRR, and nDCG have become core tools for evaluating RAG systems. However, note that these metrics essentially measure "the degree of match between system-returned results and the annotated relevance set," not absolute clinical evidence completeness. When the annotation set itself is incomplete, the metrics reflect the relative consistency between the system and the annotations, not the system's evidence coverage capability under real clinical needs. This is precisely the background against which the study emphasizes "annotation coverage."

Optimal Performance Under Hybrid Retrieval Configuration

Among the four retriever configurations, the selected hybrid retriever with 384-token chunk size and 20% overlap performed best: 17 of 18 queries retrieved acceptable evidence in the top 3 results. Recall@3, R-Precision, MRR@3, and nDCG@3 all reached 0.94. Cross-encoder reranking was confirmed to deliver the highest retrieval scores, consistent with previous findings in general domains—fine-grained semantic matching correction at the reranking stage is equally effective for specialized medical text.

Metric	Value
Recall@3	0.94
R-Precision	0.94
MRR@3	0.94
nDCG@3	0.94

The generation side was evaluated separately under a 512-token index with K=1 to 5, using DeepEval's claim-level metrics with deepseek^Ad-v4-pro as the LLM judge, supplemented by traditional BLEU/ROUGE metrics. This design of "evaluating retrieval and generation separately" avoids generation quality fluctuations masking real problems in the retrieval stage.

Cross-Encoder Reranking is a commonly used technique at the reranking stage of retrieval pipelines. Unlike bi-encoders, which encode queries and documents separately and compute vector similarity, cross-encoders concatenate the query and candidate document and feed them jointly into a Transformer to directly output a relevance score. This approach captures fine-grained semantic interactions between query and document, yielding higher accuracy but greater computational cost, so it is typically used only to rerank a small number of candidates (e.g., top 50) returned by the recall stage. In this paper, cross-encoder reranking is performed after hybrid retrieval (dense vectors + BM25) recall—a typical "coarse recall, fine rerank" architecture. BM25 captures exact keyword matches, dense vectors handle semantic recall, and the cross-encoder performs fine correction on the fused candidate set; the three complement each other to support the 0.94 retrieval metrics.

Annotation Sensitivity: Rankings Unchanged, but Evidence Identification Changed

The study's most noteworthy finding lies in the annotation expansion experiment. When ground truth expanded from 25 evidence snippets to 37 and acceptable chunks from 36 to 51, retrieval rankings did not budge. On the surface this demonstrates system robustness, but the deeper implication is: the original annotations may have missed a large amount of evidence that should have been deemed relevant. If evaluated only with initial annotations, some chunks retrieved by the system would be judged irrelevant; after annotation expansion, these chunks were re-identified as valid evidence. The scores didn't change because the ranking order was stable, but the absolute count of "relevant" items changed.

This points directly to a methodological problem: ground truth in RAG retrieval evaluation is not naturally objective, and its coverage systematically affects our judgment of system capability. Especially in clinical domains, the boundaries of evidence snippets are often fuzzy, and different annotators may have differing understandings of "sufficient evidence."

R-Precision is a metric in information retrieval evaluation, defined as the ratio of relevant documents among the top R positions of retrieval results, where R equals the total number of relevant documents actually present in the annotations for that query. For example, if a query has 3 relevant evidence chunks (R=3) and 2 of the top 3 returned results are relevant, then R-Precision is 2/3≈0.67. The characteristic of this metric is that it automatically adjusts the cutoff position based on the number of relevant documents per query, so unlike Recall@K with its fixed K value, it more fairly reflects precision under the condition of "recall exactly as many as should be recalled." In this paper, R-Precision reached 0.94, consistent with Recall@3, MRR@3, and nDCG@3, indicating the system not only meets recall quantity targets but also performs stably in ranking quality. But as the annotation expansion experiment shows, R-Precision's denominator R comes directly from annotations, and changes in annotation coverage alter R's value, so this metric is equally sensitive to annotation quality.

Limitations of Small Samples and Retrieval-Informed Annotation

The authors acknowledge two key limitations. First, the development set of 18 documents and 18 queries is very small, with limited statistical power, and conclusions require independent validation. Second, the annotation process was "retrieval-informed"—annotators may have been influenced by retrieval results, posing a risk of circular reasoning. The study therefore explicitly calls for independent assessment of evidence sufficiency by radiologists, rather than relying solely on snippets returned by the retrieval system to define relevance.

This has practical implications for the development workflow of clinical RAG systems: domain experts should be introduced at the annotation stage, and annotation strategies should be blind to retrieval results. Otherwise, the system may achieve high scores on a standard shaped by its own outputs while exposing insufficient evidence coverage in real clinical scenarios.

Retrieval-Informed Annotation refers to annotators having already been exposed to or referenced the retrieval system's outputs when annotating relevance, which may unconsciously bias the annotation set toward content the system can already retrieve. The risk of this practice is circular reasoning: the system is evaluated on a standard shaped by its own outputs, so scores are naturally high, yet it cannot reflect the system's evidence coverage blind spots on unseen new queries or in real clinical scenarios. In RAG evaluation, the ideal approach is a "blind to retrieval results" annotation strategy—annotators judge independently which snippets constitute sufficient evidence based solely on original documents and queries, and then another group of experts validates annotation consistency. The authors explicitly identify this limitation and call for radiologists to independently assess evidence sufficiency, precisely to break the feedback loop between annotation and the retrieval system and ensure the independence of ground truth.

Implications for RAG Evaluation Practice

Although limited in scale, this study's proposed evidence-aware evaluation protocol is worth adopting. It moves relevance from the document level down to the evidence snippet level and requires all overlapping chunks containing equivalent evidence to be counted as relevant, reducing interference from chunking strategy on evaluation results. For teams building RAG systems in high-stakes domains like medicine and law, there are at least two actionable recommendations:

1. Report annotation coverage: When publishing metrics like Recall@K, also specify the scale and source of ground truth so readers can judge the confidence interval of the scores.
2. Conduct annotation sensitivity analysis: Deliberately expand or contract annotations on small development sets and observe the magnitude of changes in rankings and scores as auxiliary evidence of system robustness.

Retrieval scores are never just a function of system capability; they are equally a function of annotation quality. Making this dependency explicit in the evaluation protocol is more meaningful than simply pursuing higher nDCG.

One-line Summary

A clinical RAG study finds retrieval scores highly dependent on annotation coverage—rankings unchanged after annotation expansion but relevant evidence count increases.

Paragraph Summary

An empirical clinical RAG study on breast lesion management literature reveals that retrieval scores are highly dependent on annotation coverage. The study built a benchmark of 18 documents and 18 queries; a hybrid retriever with 384-token chunks and 20% overlap hit evidence in the top 3 for 17 queries, with all four retrieval metrics reaching 0.94. The key finding: after annotations expanded from 25 evidence snippets to 37, retrieval rankings did not budge, but acceptable chunks increased from 36 to 51, indicating the original annotations may have missed substantial relevant evidence. The study also notes the 18-document small sample and retrieval-informed annotation as main limitations, calls for independent assessment by radiologists, and recommends that RAG evaluations report annotation coverage and conduct annotation sensitivity analysis.