LLM Document Information Extraction Under OCR Noise: A Comprehensive Benchmark

Large language models (LLMs) are increasingly being used to extract structured information from documents, yet their performance under real-world OCR noise conditions has rarely been studied systematically. A recent paper published on arXiv, titled From Pixels to Pairs, provides a rigorous benchmark evaluation of this problem — revealing a significant gap between clean-text lab conditions and real-world deployment.

The Core Problem

Key-Value Pair (KVP) extraction is a foundational task in document intelligence — identifying structured "field name–field value" information from invoices, forms, and receipts. In recent years, open-source instruction-tuned LLMs have been seen as promising candidates to replace traditional, layout-aware specialized systems with a single general-purpose model.

But a critical reality is often overlooked: before documents reach an LLM, they typically go through OCR (Optical Character Recognition) to convert them into text. OCR is imperfect — recognition errors, missing characters, and corrupted numbers all introduce noise. As the paper notes, most existing evaluations are conducted under "clean text" (Gold-text) conditions, which are severely disconnected from real-world deployment scenarios. The research team therefore designed a unified evaluation protocol specifically designed to examine the performance gap between clean text and noisy OCR inputs.

OCR (Optical Character Recognition) technology analyzes pixels in scanned images or photos and converts them into machine-readable text. Mainstream engines include commercial solutions (such as Google Cloud Vision and Azure OCR) and open-source solutions (such as Tesseract, PaddleOCR, and EasyOCR, all used in this paper). OCR quality is affected by multiple factors: document print quality, scan resolution, font type, layout complexity, and lighting conditions. In financial documents, historical archives, or low-quality mobile photos, OCR error rates can reach 5%–20%, with common issues including visually similar character substitutions (e.g., "0" vs. "O", "1" vs. "l"), multi-character merging or splitting errors, loss of decimal points and currency symbols, and disordered line sequences in multi-column layouts. These noise artifacts propagate to downstream LLMs in unpredictable ways, posing serious challenges to inference tasks that rely on contextual semantics.

Experimental Design: Five Models, Three Datasets, Three OCR Engines

The scope of this benchmark is substantial. On the model side, the study selects representative open-source decoder-only models: Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek — a cross-section of today's mainstream open-source landscape.

For datasets, the paper uses three classic benchmarks in the document extraction domain: FUNSD (form understanding), CORD (receipt extraction), and SROIE (scanned receipt information extraction). To simulate real-world input degradation, the researchers used not only human-annotated Gold-text, but also the actual outputs from three widely used OCR engines: PaddleOCR, EasyOCR, and Tesseract.

This cross-factorial design of "model × dataset × input quality" allows the study to isolate the independent effects of three variables under consistent conditions: input quality, model architecture, and prompting strategy. This is the paper's key methodological contribution — avoiding the confounded variables and hard-to-reproduce conclusions that have plagued prior evaluations.

FUNSD, CORD, and SROIE are three landmark benchmark datasets in document information extraction, each targeting different scenarios. FUNSD (Form Understanding in Noisy Scanned Documents) contains scanned form images annotated with text regions, semantic field roles, and inter-field relationships, with relatively high noise levels. CORD (Consolidated Receipt Dataset) focuses on retail receipts, featuring diverse receipt images from Indonesia with fine-grained annotations down to item names, prices, and tax amounts. SROIE (Scanned Receipts OCR and Information Extraction) is the ICDAR 2019 competition dataset, targeting English scanned receipts and requiring extraction of four key fields: merchant name, date, address, and total amount. Their differences in layout structure, linguistic complexity, and field types make cross-dataset evaluation conclusions more generalizable.

Key Findings: Strong on Clean Text, Significant Degradation Under Noise

The results reveal a clear duality. When the input is high-quality text, modern LLMs demonstrate strong semantic extraction capabilities, in some cases approaching the performance of supervised, layout-aware specialized systems. This confirms the potential of general-purpose large models for structured understanding.

However, once OCR noise is introduced, performance drops substantially. More intriguingly, as input degradation increases, the performance gap between different models actually narrows. In other words, models that lead on clean text lose their advantage in noisy environments.

The paper attributes extraction performance to two dominant factors: semantic reasoning ability over text and preservation of text fidelity under OCR noise. Larger models bring gains on clean text, but these gains gradually disappear with noisy inputs — at which point OCR quality itself becomes the decisive factor. This means that blindly scaling up model parameters cannot solve real-world deployment problems; the quality bottleneck at the input stage is often more critical.

Recurring Failure Patterns

The study also identifies several typical error patterns with strong practical implications for engineering:

• Key-value misalignment: The model pairs a field name with the wrong value, especially after layout information is lost.
• Hallucination: The model generates content that does not exist in the document, which is particularly pronounced when noise causes information loss.
• Numeric corruption: Numerical fields such as amounts and dates are corrupted by OCR errors and then reproduced as-is in model outputs — extremely harmful for financial documents.

These failure patterns remind developers that in production environments, overall accuracy alone is not enough; these high-risk error types require targeted monitoring.

Hallucination is a systematic defect in large language models where the model generates content that is factually incorrect or simply absent from the original input, yet presents it with high confidence. In document extraction tasks, the causes of hallucination differ somewhat from general QA scenarios: when OCR noise leaves original field information incomplete or semantically ambiguous, the model tends to fill in the gaps with "plausible" values common in training data — for example, completing a corrupted date with a correctly formatted but incorrect date, or substituting a missing tax rate with a common default value. This "harmful auto-correction" behavior is difficult to distinguish from correct extraction at the output level, yet can have serious consequences in high-precision domains like finance and law. Detecting hallucinations therefore requires dedicated evaluation mechanisms independent of accuracy metrics — such as comparing model outputs against the literal content of the original OCR text.

Implications for Real-World Deployment

The most valuable conclusion of this research is its clear identification of the gap between "clean-text evaluation" and "real-world deployment." Many approaches that shine on paper leaderboards see their effectiveness collapse once connected to a real OCR pipeline.

The paper advocates for joint optimization: rather than improving any single component in isolation, teams should simultaneously improve OCR quality, structured reasoning capabilities, and LLM semantic modeling. For teams building document intelligence systems, this means that when evaluating solutions, you should not focus solely on model leaderboard rankings — instead, treat the entire data pipeline, from pixels to key-value pairs, as a whole to be evaluated and optimized together.

With the rise of multimodal large models, it may eventually be possible to bypass the standalone OCR step and extract information end-to-end directly from images. But under the current text-LLM-dominant architecture, this benchmark provides a pragmatic and sobering reference point.