ExtractBench Pro Reveals: Complex Chart Reasoning Remains a Weakness for Frontier Models

Real-world documents often embed information-dense charts. Humans don't just "read" the values in a chart—they reason over them and cross-reference them with other complex elements like tables to complete knowledge tasks. However, this kind of integrated reasoning remains a major challenge for current AI models. The recently released benchmark ExtractBench Pro is specifically designed for this scenario, and the results show that even the most frontier models perform poorly when it comes to perfectly solving these complex extraction tasks.

Why Is Chart + Document Reasoning So Difficult?

Charts do not exist in isolation. In a real report or research document, charts are typically tightly coupled with body text, data tables, annotations, and other elements. Human readers can naturally move between these elements, extract key values, understand trends, and perform logical reasoning with context. For example, in a financial report, a reader may need to read quarterly revenue from a bar chart, compare it with cost data in a table, and finally calculate the profit margin.

This multimodal, multi-step reasoning process places extremely high demands on AI models: the model must not only accurately identify chart types, axes, legends, and data points, but also understand the relationships expressed by the chart and align them with other information in the document. ExtractBench Pro was built precisely to systematically evaluate this capability.

A Multimodal Large Language Model (MLLM) refers to a model that can process multiple modalities such as text and images simultaneously and generate text output. Typical examples include GPT-4V, Gemini, Claude 3, and others. Such models usually use a visual encoder (such as ViT) to convert images into visual tokens, which are then fed together with text tokens into the language model for joint reasoning. However, the resolution limits of the visual encoder and token compression can lose fine-grained information in charts (such as small-font tick marks and dense data points), while the language model part lacks explicit modeling of spatial relationships, causing the model to be prone to errors in tasks requiring precise numerical reading and cross-element alignment. This is precisely the architectural reason why chart + document reasoning becomes a weakness.

ExtractBench Pro: A Benchmark Specifically for Evaluating Complex Extraction

According to the publisher's introduction on Twitter, ExtractBench Pro is a brand-new benchmark designed to capture the complex "chart + document" reasoning tasks found in real-world documents. It goes beyond simple chart value perception and requires models to reason while extracting information, and to coordinate with other complex elements (such as tables) to solve knowledge-based tasks.

The launch of this benchmark stems from an observation: existing benchmarks often simplify document structure, whereas the complexity of real documents is far higher. ExtractBench Pro attempts to bridge this gap and provide a stricter test of models' information extraction capabilities in real-world scenarios.

In the AI field, a benchmark refers to a standardized dataset and evaluation protocol used to compare the capabilities of different models on specific tasks. A high-quality benchmark usually includes diverse real samples, clear evaluation metrics (such as accuracy, F1, and exact match rate), and a reproducible evaluation process. Unlike training sets, the core value of a benchmark lies in preventing data contamination and reflecting a model's generalization ability. The "Pro" suffix in ExtractBench Pro suggests a significant increase in task complexity or scale compared with the base version, while "Extract" emphasizes that its evaluation focus is information extraction rather than simple chart question answering.

How Do Frontier Models Perform?

The publisher admitted: "Even the best frontier models struggle to perfectly solve these complex extraction tasks on documents." This conclusion is not surprising. Although current multimodal large models have made progress on tasks such as image captioning and simple chart question answering, their error rates remain high in complex scenarios requiring multi-step reasoning and cross-element integration.

For example, a model may correctly identify a data point in a chart but fail to associate it with the corresponding item in a table; or it may read the value correctly but make logical deviations during reasoning. The evaluation results of ExtractBench Pro reveal these weaknesses and also point the way for future model improvements.

From Benchmark to Application: LlamaParse's Positioning

In addition to the benchmark, the publisher also mentioned LlamaParse—a tool aimed at large-scale complex extraction needs. This implies that ExtractBench Pro not only has academic evaluation value but is also closely connected to practical applications. For enterprises that need to process large volumes of complex documents, how to reliably extract information from charts and tables is a real pain point. LlamaParse may provide an engineering solution, while ExtractBench Pro can be used to measure the effectiveness of such tools.

LlamaParse is a document parsing service launched by the LlamaIndex team, specifically designed to convert unstructured documents such as PDFs and images into structured text or Markdown. It is especially good at handling complex layouts containing tables, charts, and formulas. Its technical approach usually combines OCR, layout analysis models, and vision-language models to perform region-based recognition and structured reconstruction of documents. Compared with general OCR tools, LlamaParse places more emphasis on preserving the semantic structure of documents and the relationships between elements. This corresponds closely to the "cross-element coordinated reasoning" capability evaluated by ExtractBench Pro, and also explains why the publisher mentioned the two together.

Significance for the AI Community

The emergence of ExtractBench Pro reminds us that document understanding is far from solved. As model capabilities improve, evaluation benchmarks also need to continuously evolve to reflect real-world complexity. For researchers, this benchmark provides a new testing platform; for developers, it points out the limitations of current systems; for users, it means they need to remain cautious about "complex document processing" capabilities when choosing tools.

In the future, we look forward to seeing more models make breakthroughs on ExtractBench Pro, and we also hope that related tools can truly help people efficiently mine knowledge from massive document collections.

One-line Summary

ExtractBench Pro reveals that frontier models still fall short in joint chart and document reasoning extraction.

Paragraph Summary

ExtractBench Pro is a newly released benchmark specifically designed to evaluate models' ability to perform complex information extraction and reasoning by combining charts, tables, and body text in real documents. The article points out that charts in real documents are often tightly coupled with tables, annotations, and other elements. Humans can naturally move between them to complete multi-step reasoning, whereas current multimodal large models can recognize individual data points but struggle to complete cross-element alignment and logical integration. The publisher admitted that even the most frontier models struggle to perfectly solve such tasks. This benchmark not only has academic evaluation value but also echoes engineering document parsing tools such as LlamaParse, reminding the community that document understanding is far from solved and that evaluation benchmarks need to continuously evolve to reflect real-world complexity.