When Figures Disagree with Text: Blind Spots of Multimodal Models in Paper Review

Multimodal models are severely lacking in autonomously discovering and verifying figure-text contradictions in complete papers.
A study systematically evaluated multimodal models' ability to autonomously verify visual evidence in scientific papers. It selected 31 complete manuscripts, each with a hidden figure-text inconsistency, and had six models review the PDFs under a unified prompt. Of 1,445 review comments, 60.6% mentioned flawed figures, but only 3.3% were accepted as genuinely diagnosing contradictions. Key findings: merely naming the figure (without stating the flaw) raised diagnosis rates from 4.3% to 20.4%, while consistency instructions or visual hints were ineffective; text-only review could not diagnose contradictions existing only in rendered figures, and combining rendered pages with figure names reached 15.8%; a baseline letting models choose figures achieved only a 4.8% diagnosis rate. The study shows models struggle greatly with deciding "what to check," and multimodal document understanding evaluation should focus on models' ability to proactively propose verification targets.
Figures in scientific papers are key evidence supporting arguments, so reviewing a paper requires not only describing figure content but also checking whether figures and body text are consistent. However, existing figure-text verification benchmarks typically provide the figure, claim, or known inconsistency directly, which masks a more fundamental question: can a model autonomously find figures worth checking in a complete paper, associate visual elements with conflicting claims, and identify contradictions?
A recent study systematically evaluated this and revealed significant shortcomings of multimodal models on autonomous visual evidence verification tasks.

Design of the Autonomous Verification Task
The study selected 31 complete manuscripts, each containing a figure-text inconsistency annotated by PRISMM-Bench, with the issue hidden from the model. Six multimodal models reviewed each PDF under a unified neutral prompt, after which researchers added additional inputs one by one for comparison.
Of a total of 1,445 open-ended review comments, 60.6% mentioned the flawed figure, but the scoring rubric accepted only 3.3% of cases as genuinely diagnosing the contradiction. Because blind review found the scoring rubric to be lenient, the researchers treated these as "rubric acceptance rates" and mainly compared relative changes across conditions rather than absolute values.
PRISMM-Bench is a benchmark dataset for evaluating consistency between scientific figures and text. Its annotations include the types of inconsistency between figures and body-text claims (such as data contradictions, misstated trends, labeling errors, etc.) as well as the specific figures and claim snippets involved. The benchmark is characterized by fine-grained inconsistency annotations and is typically used to test a model's ability to judge consistency given known figures and claims. This study further uses PRISMM-Bench annotations as "hidden answers" to test whether a model can autonomously discover these annotated contradictions in complete papers, thereby advancing evaluation from "judging consistency given figures and claims" to "autonomously locating and verifying inconsistencies in real documents."
Key Finding: Naming the Figure Substantially Improves Diagnosis Rates
In a separate controlled experiment, merely telling the model the figure's name (without stating that the figure was flawed) raised the rubric-accepted diagnosis rate from 4.3% to 20.4%. By contrast, providing consistency instructions or coarse visual hints had no effect.
When the contradiction existed only in the rendered figure, text-only review produced no accepted diagnoses, whereas reviewing the rendered page and naming the figure achieved a diagnosis rate of 15.8%.
A baseline method that let the model choose which figures to check on its own (ranked by centrality) achieved only a 4.8% diagnosis rate, indicating that models have great difficulty deciding "what to check" and that the selection rules tested in the study failed to solve this problem.
Centrality in graph theory and network analysis measures the importance of nodes, with common metrics including degree centrality, betweenness centrality, closeness centrality, and others. In the context of this study, ranking figures by centrality may refer to ordering them by the number of times a figure is cited in the paper, its frequency of appearance in the body text, or its degree of association with other figures/paragraphs. This baseline method assumes that figures cited more often or more central are more likely to contain problems, but the results show a diagnosis rate of only 4.8%, indicating that this heuristic based on citation frequency cannot effectively replace the model's autonomous judgment of "what to check."
Implications for Multimodal Document Understanding
The study emphasizes that multimodal document understanding should evaluate whether a model can proactively propose "what needs verification," rather than merely answering questions about a given figure. Current models still have clear weaknesses in autonomously discovering and verifying visual evidence, especially in complex scenarios requiring cross-modal reasoning.
This finding has important reference value for scientific paper review, automated fact-checking, and the design of multimodal AI systems. Future evaluation benchmarks need to be closer to real review workflows and examine models' exploration and judgment capabilities under unguided conditions.
Multimodal Document Understanding refers to a model's ability to simultaneously process multiple modalities such as text, images, tables, and charts, and to extract, integrate, and reason over information from complete documents. Unlike traditional visual question answering (VQA), visual elements in document understanding tasks (such as figures) have complex referential and dependency relationships with text, and models need to understand layout structure, cross-modal reference, and logical consistency between figures and text. Current multimodal large models perform relatively well on question-answering tasks given figures and questions, but still show significant gaps in scenarios requiring proactive exploration of documents, discovery of potential contradictions, and autonomous determination of verification targets. This highlights the necessity of a paradigm shift in evaluation from "passive answering" to "active review."
One-line Summary
Multimodal models are severely lacking in their ability to autonomously discover and verify figure-text contradictions in complete papers.
Paragraph Summary
A study systematically evaluated multimodal models' ability to autonomously verify visual evidence in scientific papers. The study selected 31 complete manuscripts, each containing a hidden figure-text inconsistency, and six models reviewed the PDFs under a unified prompt. Of 1,445 review comments, 60.6% mentioned the flawed figure, but only 3.3% were accepted by the scoring rubric as genuinely diagnosing the contradiction. The key finding is that merely naming the figure (without stating the flaw) raised the diagnosis rate from 4.3% to 20.4%, while consistency instructions or visual hints were ineffective; text-only review could not diagnose contradictions existing only in rendered figures, and combining rendered pages with figure names reached 15.8%; a baseline method letting the model choose which figures to check achieved only a 4.8% diagnosis rate. The study shows that models have great difficulty deciding "what to check," and multimodal document understanding evaluation should focus on models' ability to proactively propose verification targets.
Related articles

Gemini 4 Neon Review: Benchmark Scores Rival Carbon, API Pricing and Quota Breakdown
Google Gemini 4 Neon is now live, benchmark scores closely tracking Carbon, API pricing 3/15 at twice Hydrogen's rate. This article compiles Bilibili creator testing: Neon can replace 90% of Carbon scenarios, AntiGravity quota consumption is only one-third of Carbon, plus subscription channel restrictions and 2API ban risks.

Gemini 4 Helium Coming Soon? Signals of Google Loosening Subscription Restrictions
Google team member hints releases will continue; community speculates Gemini 4 Helium (Flash Lite) may be next. Subscription ban controversy gets a response as product lead says "Soon," suggesting restrictions may be lifted. This article covers Gemini developments, OpenAI Coding Agent and risk management, Doubao real-time audio-video API, and compute business dynamics.

AI-Assisted React Development in Practice: Optimizing a Video App and Handling Errors
A 14-year veteran developer demonstrates AI-assisted React video app optimization: compressing video from 80MB to 5MB (~94%) via ImageKit, plus reliable error handling and retry mechanisms. Includes planning, code understanding, and verification methods for working with AI.