[KongchangAI]
· 1 min read· 955 words

When Figures Disagree with Text: Blind Spots of Multimodal Models in Paper Review

When Figures Disagree with Text: Blind Spots of Multimodal Models in Paper Review

Multimodal models are severely lacking in autonomously discovering and verifying figure-text contradictions in complete papers.

A study systematically evaluated multimodal models' ability to autonomously verify visual evidence in scientific papers. It selected 31 complete manuscripts, each with a hidden figure-text inconsistency, and had six models review the PDFs under a unified prompt. Of 1,445 review comments, 60.6% mentioned flawed figures, but only 3.3% were accepted as genuinely diagnosing contradictions. Key findings: merely naming the figure (without stating the flaw) raised diagnosis rates from 4.3% to 20.4%, while consistency instructions or visual hints were ineffective; text-only review could not diagnose contradictions existing only in rendered figures, and combining rendered pages with figure names reached 15.8%; a baseline letting models choose figures achieved only a 4.8% diagnosis rate. The study shows models struggle greatly with deciding "what to check," and multimodal document understanding evaluation should focus on models' ability to proactively propose verification targets.

Figures in scientific papers are key evidence supporting arguments, so reviewing a paper requires not only describing figure content but also checking whether figures and body text are consistent. However, existing figure-text verification benchmarks typically provide the figure, claim, or known inconsistency directly, which masks a more fundamental question: can a model autonomously find figures worth checking in a complete paper, associate visual elements with conflicting claims, and identify contradictions?

A recent study systematically evaluated this and revealed significant shortcomings of multimodal models on autonomous visual evidence verification tasks.

rss source: When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Pa

Design of the Autonomous Verification Task

The study selected 31 complete manuscripts, each containing a figure-text inconsistency annotated by PRISMM-Bench, with the issue hidden from the model. Six multimodal models reviewed each PDF under a unified neutral prompt, after which researchers added additional inputs one by one for comparison.

Of a total of 1,445 open-ended review comments, 60.6% mentioned the flawed figure, but the scoring rubric accepted only 3.3% of cases as genuinely diagnosing the contradiction. Because blind review found the scoring rubric to be lenient, the researchers treated these as "rubric acceptance rates" and mainly compared relative changes across conditions rather than absolute values.

PRISMM-Bench is a benchmark dataset for evaluating consistency between scientific figures and text. Its annotations include the types of inconsistency between figures and body-text claims (such as data contradictions, misstated trends, labeling errors, etc.) as well as the specific figures and claim snippets involved. The benchmark is characterized by fine-grained inconsistency annotations and is typically used to test a model's ability to judge consistency given known figures and claims. This study further uses PRISMM-Bench annotations as "hidden answers" to test whether a model can autonomously discover these annotated contradictions in complete papers, thereby advancing evaluation from "judging consistency given figures and claims" to "autonomously locating and verifying inconsistencies in real documents."

Key Finding: Naming the Figure Substantially Improves Diagnosis Rates

In a separate controlled experiment, merely telling the model the figure's name (without stating that the figure was flawed) raised the rubric-accepted diagnosis rate from 4.3% to 20.4%. By contrast, providing consistency instructions or coarse visual hints had no effect.

When the contradiction existed only in the rendered figure, text-only review produced no accepted diagnoses, whereas reviewing the rendered page and naming the figure achieved a diagnosis rate of 15.8%.

A baseline method that let the model choose which figures to check on its own (ranked by centrality) achieved only a 4.8% diagnosis rate, indicating that models have great difficulty deciding "what to check" and that the selection rules tested in the study failed to solve this problem.

Centrality in graph theory and network analysis measures the importance of nodes, with common metrics including degree centrality, betweenness centrality, closeness centrality, and others. In the context of this study, ranking figures by centrality may refer to ordering them by the number of times a figure is cited in the paper, its frequency of appearance in the body text, or its degree of association with other figures/paragraphs. This baseline method assumes that figures cited more often or more central are more likely to contain problems, but the results show a diagnosis rate of only 4.8%, indicating that this heuristic based on citation frequency cannot effectively replace the model's autonomous judgment of "what to check."

Implications for Multimodal Document Understanding

The study emphasizes that multimodal document understanding should evaluate whether a model can proactively propose "what needs verification," rather than merely answering questions about a given figure. Current models still have clear weaknesses in autonomously discovering and verifying visual evidence, especially in complex scenarios requiring cross-modal reasoning.

This finding has important reference value for scientific paper review, automated fact-checking, and the design of multimodal AI systems. Future evaluation benchmarks need to be closer to real review workflows and examine models' exploration and judgment capabilities under unguided conditions.

Multimodal Document Understanding refers to a model's ability to simultaneously process multiple modalities such as text, images, tables, and charts, and to extract, integrate, and reason over information from complete documents. Unlike traditional visual question answering (VQA), visual elements in document understanding tasks (such as figures) have complex referential and dependency relationships with text, and models need to understand layout structure, cross-modal reference, and logical consistency between figures and text. Current multimodal large models perform relatively well on question-answering tasks given figures and questions, but still show significant gaps in scenarios requiring proactive exploration of documents, discovery of potential contradictions, and autonomous determination of verification targets. This highlights the necessity of a paradigm shift in evaluation from "passive answering" to "active review."

One-line Summary

Multimodal models are severely lacking in their ability to autonomously discover and verify figure-text contradictions in complete papers.

Paragraph Summary

A study systematically evaluated multimodal models' ability to autonomously verify visual evidence in scientific papers. The study selected 31 complete manuscripts, each containing a hidden figure-text inconsistency, and six models reviewed the PDFs under a unified prompt. Of 1,445 review comments, 60.6% mentioned the flawed figure, but only 3.3% were accepted by the scoring rubric as genuinely diagnosing the contradiction. The key finding is that merely naming the figure (without stating the flaw) raised the diagnosis rate from 4.3% to 20.4%, while consistency instructions or visual hints were ineffective; text-only review could not diagnose contradictions existing only in rendered figures, and combining rendered pages with figure names reached 15.8%; a baseline method letting the model choose which figures to check achieved only a 4.8% diagnosis rate. The study shows that models have great difficulty deciding "what to check," and multimodal document understanding evaluation should focus on models' ability to proactively propose verification targets.

#multimodal models#figure-text consistency#scientific paper review#visual evidence verification#document understanding#PRISMM-Bench
Share:

Related articles