The "Claim Smoothing" Phenomenon in LLM Research Reports: Quantitative Evaluation with SmoothScore

Large language models (LLMs) are widely used for deep research—reading large volumes of documents and generating insights. However, a new study finds that long-form research reports generated by LLMs often read like confident, synthesized analyses while failing to accurately reflect the underlying documents. The researchers call this failure "claim smoothing" and propose SmoothScore to quantify it.

What Is Claim Smoothing?

When synthesizing information from different sources, LLMs tend to compress diverse and potentially conflicting sources into a single narrative. In the process, the model discards the "jaggedness" of the evidence in the corpus and delivers more decisive conclusions than the citations can support. This effect is claim smoothing.

The researchers note that deep research—or any application that uses LLMs to read large collections of documents and generate insights—requires synthesizing information from different sources. Smoothing causes the generated reports to appear coherent and confident while actually being disconnected from the evidentiary basis.

In natural language processing and argument mining, a "claim" typically refers to a declarative statement that can be verified or refuted, such as "a certain drug significantly reduced mortality in clinical trials." It differs from questions, instructions, or purely emotional expressions, and constitutes the smallest semantic unit of a research report's conclusions. In long-form reports generated by LLMs, key conclusive claims are often scattered across paragraphs and require dedicated extraction for evidence alignment and evaluation. SmoothScore scores at the level of claims rather than entire reports or individual sentences, because only claims directly carry the judgment of "whether a conclusion is supported by evidence." Understanding this technical meaning of claim helps grasp why smoothing is defined as "overconfidence at the claim level" rather than simple exaggerated wording.

SmoothScore: Quantifying the Degree of Smoothing

To quantify this effect, the researchers introduce a claim-level scoring pipeline called SmoothScore. The pipeline consists of three steps:

1. Extract key conclusive claims: Extract the key conclusive claims from the model-generated report.
2. Find relevant evidence and cluster it: For each claim, find relevant evidence and cluster this evidence along high-level aspects.
3. Score: Score each claim based on the evidence to quantify the degree of smoothing performed by the agent.

Through this pipeline, the researchers can objectively measure the extent to which the model exaggerates the certainty of its conclusions.

Evidence clustering is a key step in the SmoothScore pipeline. When multiple sources discuss the same claim from different angles, the raw evidence snippets are often highly heterogeneous in wording, granularity, and stance. Clustering merges these snippets by "high-level aspects"—for example, grouping evidence about a policy's effects into thematic clusters such as "economic impact," "social equity," and "implementation cost." This serves two purposes: first, it avoids mistakenly counting large amounts of duplicate or near-duplicate evidence as independent support, which would artificially inflate a claim's credibility; second, it allows scoring to identify contradictions within the evidence—if the same claim receives conflicting evidence across different aspects, the degree of smoothing should be judged higher. Clustering quality directly affects the reliability of SmoothScore, so researchers typically need to combine semantic similarity with manual verification to build stable evidence clusters.

Experimental Findings: Agents Consistently Produce Stronger Claims

The researchers evaluated 7 systems on 4 datasets. The results show that agents consistently produce stronger claims than the evidence supports. This means that regardless of which system or dataset is used, LLM research agents exhibit an overconfidence bias, delivering conclusions that are more absolute than the citations can support.

This finding has important cautionary implications for research work that relies on LLMs for literature reviews, market analysis, or any task requiring synthesis of multi-source information. Users need to be aware that LLM-generated reports may obscure uncertainty and contradictions in the evidence.

Here, "agent" refers to a research-oriented LLM system with multi-step reasoning and tool-calling capabilities, rather than a single-turn question-answering model. A typical deep research agent autonomously plans retrieval paths, calls search engines or document libraries, filters and reads multiple materials, and finally synthesizes a report. Compared with single-turn generation, an agent's multi-step workflow introduces more decision points: which documents to retrieve, which sources to trust, and how to trade off conflicting information. Claim smoothing is particularly prominent in agent settings because at each step the agent tends to select information that advances narrative coherence while systematically downplaying evidence that would weaken the certainty of conclusions. Therefore, the "consistently producing stronger claims" observed in the experiments is not an accidental bias of a particular model, but a structural tendency of agentic workflows.

Implications for LLM Research Applications

Claim smoothing reveals a systematic flaw in current LLMs on deep research tasks. The models tend to pursue narrative coherence at the expense of faithfully reflecting the complexity of the evidence. This can lead to misleading conclusions, especially in domains where the evidence itself is contested or uncertain.

In the future, improving LLMs' research capabilities may require developing better training objectives and evaluation metrics to encourage models to preserve the jaggedness of evidence when synthesizing information rather than over-smoothing. SmoothScore provides a quantifiable tool for this direction.