Why MCP Retrieval Service Citations Get Lost: A Deep Dive into CSR Benchmarking

When a company turns its documentation into an MCP search tool and hands it to Claude, ChatGPT, or any Agent, the retrieval side can guarantee the right sources are sent in—but it has zero visibility into what the final answer actually cites. This question of whether citations survive past the boundary is becoming a new blind spot at the intersection of RAG and Agents.

Problem Definition: Do Citations Survive Past the MCP Boundary?

Typical RAG is "retrieve once, answer once"—the retrieval service and the generation model live in the same system, so the pipeline is controllable. But once retrieval capabilities are exposed to external Agents via MCP, control splits: the retrieval server returns document chunks with source_url, and the Agent decides which ones to read, how to structure the answer, and whether to mark citations.

For documentation assistants, this creates two levels of failure. The answer is correct but has no citations, so users cannot click through to view edge cases; or the answer cites the wrong page, which looks verified but is actually more misleading. The author manually filtered 250 questions down to 127, of which 82 are answerable from a single page, 29 require cross-page synthesis, and 16 are not covered by the documentation—used to catch fabrication behavior. The corpus is FastAPI documentation, with 153 pages split into 2,085 heading-aware chunks.

The key to this experimental design is that it replicates the contract of a real hosted MCP retrieval product—even the tool naming style is aligned—rather than a toy environment.

Citation Survival Rate: More Than Just Checking for URLs

Measuring "whether there is a citation" is trickier than it seems. An answer may be full of URLs that are attributed incorrectly, or may fabricate a plausible-looking FastAPI link—the model already knows what such URLs look like.

So the author defined CSR (Citation Survival Rate), requiring every citation to pass two checks: a support check using FActScore-style atomic claim decomposition plus an ALCE-style entailment model to verify that the cited chunk actually supports the claim; and a provenance check relying on tool call logs to confirm that the URL really appeared in the content the Agent retrieved.

CSR has an important constraint on its denominator: it only counts correct sources that the Agent actually retrieved. If the Agent never found the correct page, that's a retrieval failure and is not counted in CSR. It measures only what happens to correct sources after they cross the boundary.

FActScore-style atomic claim decomposition: FActScore is a fine-grained factuality evaluation method that breaks a long text into a number of independent atomic facts, each containing only one verifiable proposition, and then checks each one for source support. This article borrows that idea to decompose the Agent-generated answer into minimal claim units, thereby determining which claim each cited chunk actually supports—avoiding the missed detection where "the whole answer appears cited, but the citation doesn't support the specific conclusion."

ALCE-style entailment model: ALCE (Attributed Language Model Evaluation) goes beyond URL matching when evaluating citations—it uses natural language inference (NLI) or entailment models to judge whether the "retrieved document chunk" semantically entails the "claim in the answer." This article uses such entailment checks for support verification: even if the URL does come from retrieval results, if the chunk content is irrelevant to the claim, the citation is not counted as surviving.

First-Round Results and the Mistake That Almost Got Published

First-round data showed that the LangGraph Agent was roughly on par with single-pass RAG (0.614 vs. 0.618), while the Agent built with the OpenAI Agents SDK scored only 0.323—appearing to be cut in half. Of the 127 answers, 66 did not use the required [url] format.

Rather than jumping to conclusions, the author read all 66 answers one by one. It turned out that most did cite sources: 51 placed the URL as a Markdown link [source](https://...) after the claim, 7 used other bracket formats, and only 8 gave no URL at all—compared with 19 such answers in the single-pass RAG baseline.

The scorer only recognized the [url] format and recorded all Markdown links as missing citations. After fixing the definition and re-scoring, the SDK Agent rose from 0.323 to 0.470. Across three repeated experiments, a smaller loss persisted reliably: −0.12 under the original scorer, −0.09 under the gpt-4o judge, and about −0.30 when only the specified format is recognized. In other words, roughly 60% of the gap came from format, and 40% was genuine citation loss. At temperature 0, no loss was detected.

A more robust finding: under its default API, the OpenAI Agents SDK ignored the required citation format in 51 of 127 answers (40%), with 53 and 46 in two repeat runs; single-pass RAG had 0, and LangGraph had 8. Any downstream system that parses citations by a fixed format—documentation widgets, source panels, guardrails, evaluations—will see a blank.

Root-Cause Investigation: Neither the Agent Loop nor the Transport Is the Culprit

The author ran five control experiments, changing only one variable at a time.

LangGraph also runs in a loop but preserved the format, and each Agent's loop count was low—about 1.2 to 1.3 retrievals per question. Temperature was not the main factor either: the SDK pipeline never set temperature, and aligning it with LangGraph's 0 reduced format drift only from 60 to 54. The transport layer was also exonerated: dumps showed that source_url arrived intact with every chunk at the model, and 98 of 118 Markdown links pointed to pages the Agent had retrieved.

The real variable was the API. The OpenAI Agents SDK calls the Responses API by default, whereas LangGraph's OpenAI integration uses Chat Completions. Switching the same SDK Agent to Chat Completions reduced format-error answers from 41 to 9; swapping in a public grounding system prompt reduced it to 20. Both held across three rounds of experiments, but neither had a significant effect on "whether to cite" itself—they fix the format.

Since both of these live on the Agent side, which the retrieval provider cannot control, the author also tested a server-side fix: tag each chunk with a short source ID (e.g., [S3fa21c]), require citations as [ID], and map the IDs back to URLs after the answer is generated. On the self-hosted image, format-error answers dropped from about 50 to about 5 per round, the SDK Agent's CSR improved by 0.10 to 0.13, and the gap with single-pass RAG disappeared. The author also notes that these citations have not yet been human-reviewed.

The difference between the Responses API and Chat Completions: These are two generation interfaces offered by OpenAI. Chat Completions is the long-standing classic interface, centered on message lists for input and output; the Responses API is a newer interface natively designed around tool calls, multi-turn state, and structured output, and the OpenAI Agents SDK uses this path by default. The two differ in implementation at layers such as system prompt injection, tool result presentation, and output format constraints, so the same model and the same prompt may exhibit different format compliance under the two pipelines. This article's control experiments exploit exactly this difference to pinpoint the API default as the main cause of format drift.

Credibility and Boundaries

The credibility of the evaluation depends on the scorer. The author calibrated the checker against 115 of their own blind judgments, and the gpt-4o judge had the highest agreement with the author (κ 0.86). On 30 blind test cases, CSR judgments agreed with human judgment in 25 to 27 cases. Each gpt-4o-mini configuration was run for three rounds, with robustness rules written down in advance: consistent sign in each round, and combined intervals excluding zero.

Before publication, the author reviewed the pipeline from the perspective of a hostile reviewer and found five bugs, four of which would originally have shipped. The format issue was the sixth, and the most embarrassing one for the author: before it was discovered, inflated numbers had already been sent to others.

The limitations are also stated candidly: a single corpus, two models, three rounds for the gpt-4o-mini configuration but only one for Claude, a small human-check sample, and use of a self-hosted MCP server image rather than anyone's production system.

Implications for Builders

If you provide RAG via MCP, your accuracy guarantee stops at the boundary. Retrieval is controllable; the Agent writing the answer is not. If citations matter to the product, you must measure what happens on the other side of the boundary.

Framework defaults are also part of the system. With the same model and the same prompt, two Agents behave very differently on citation format—because of a default setting a developer may never look at.

Measure format and existence separately. "There is no citation in the format I parse" and "there is no citation" are two different failures with different fixes. More importantly, read the output rather than only the metrics: a count of "uncited answers" concealed the fact that most answers actually contained sources. One run is not enough—single-run results for the remaining gap could fall anywhere between −0.05 and −0.16.

Finally, the retrieval side can actually fix the format: give the Agent a short source ID to copy, rather than a long URL to reformat and then map back yourself.

MCP (Model Context Protocol): An open protocol for exposing external tools, data sources, and retrieval services to AI applications and Agents in a standardized way. The retrieval side implements an MCP server, and the Agent side calls tools through an MCP client and gets structured results. The protocol itself only specifies "how to transmit," not "how to use what you receive," so whether citations are preserved happens entirely on the Agent side—which is the origin of the article's phrase "accuracy guarantees stop at the boundary."

One-line Summary

Citation format drift past the MCP boundary was misjudged as citation loss; the default API is the main cause, and short source IDs can fix it.

Paragraph Summary

The article studies the new problem of "whether citations survive past the boundary" after exposing RAG retrieval capabilities to external Agents via MCP. The author builds a 127-question evaluation set from FastAPI documentation and defines the CSR (Citation Survival Rate) metric, checking both whether citations are supported by retrieved content and whether the URL actually appears in tool call logs. First-round results showed the OpenAI Agents SDK had a CSR of only 0.323, but reading answer by answer revealed that most "uncited" cases were actually format differences such as Markdown links; after fixing the scoring, it rose to 0.470. Control experiments showed that the Agent loop, temperature, and transport layer were not culprits—the real variable was the Responses API used by default by the SDK; after switching to Chat Completions, format errors dropped from 41 to 9. A server-side short source ID scheme further reduced format errors from about 50 to about 5 and raised CSR by 0.10–0.13.