Why VLMs Struggle to Read Forms: Inside LlamaParse's Form Parsing Model

Form documents may look simple, but they're one of the most reliably challenging scenarios for vision-language models (VLMs). Whether it's a scanned image, handwritten annotations, or a densely packed page with over a hundred fields, general-purpose document parsing solutions consistently fall short on accuracy. The LlamaIndex team recently released a model specifically designed for form reading and published the technical thinking behind it. This article breaks down the core challenges of form parsing and how they're being addressed.

Why Forms Are a Hard Problem for VLMs

Most document parsing tools target Markdown as their output format, which works well for body text, headings, and lists. But the structure of forms is far more complex than Markdown can express.

According to the LlamaIndex team, a capable form parser must maintain consistent type definitions across different elements: checkboxes, textboxes, labels, signature fields, and more — none of which can simply be collapsed into a line of plain text. If you just convert a form to Markdown, you lose critical information like whether a checkbox is checked, or which label corresponds to which input field.

In other words, the value of a form isn't in "what text it contains" — it's in "what value each field holds and how fields relate to each other." This demands a structured type system in the parser, not just raw text extraction.

Vision-language models (VLMs) are multimodal large models capable of understanding both images and text. Notable examples include GPT-4o, Claude 3, and Gemini. They work by encoding images as visual tokens alongside text tokens for joint reasoning. However, VLMs are primarily trained on natural images and web documents, which leaves them poorly suited for the "high-density structured layouts" found in forms. Spatial relationships (which label maps to which input field), element states (is a checkbox checked? is a signature field filled?), and field groupings that span rows and columns all require precise layout understanding — not semantic inference — which is exactly where general-purpose VLMs fall short.

Complexity and the Near-100% Accuracy Requirement

Another challenge with forms is that their complexity ceiling is extremely high, while the tolerance for errors is extremely low.

LlamaIndex notes that real-world forms can be scanned documents, may include handwritten annotations, and can pack more than 100 fields onto a single page. These factors combined place severe demands on a model's recognition capabilities.

More critically, form parsing use cases typically require accuracy approaching 100%. In finance, legal, healthcare, and government applications — the most common form-heavy industries — a single field error can directly cause downstream process failures or compliance violations. This is completely different from the "good enough to read" standard of general document parsing — form parsing leaves almost no room for mistakes.

This also explains why general-purpose VLMs underperform on form tasks: they're optimized for breadth, while forms require highly targeted models that remain robust in edge cases.

Grounding and Attribution: Where Did That Value Come From?

Beyond extracting values, there's another capability that's easy to overlook but critically important: grounding and attribution.

As the team describes it, a capable form parser must not only output field values — it must also precisely identify where each value appears in the source document. This traceability is nearly non-negotiable in high-stakes scenarios: when a human needs to review an extracted result, being able to jump directly to the corresponding region in the original document dramatically reduces the verification burden.

For audit trails, compliance workflows, and human-in-the-loop systems, "where in the document did this value come from" is just as important as "what is this value." Extracted results without attribution are hard to trust in serious business contexts.

Grounding & Attribution in the document AI field refers to establishing a verifiable mapping between extracted results and specific regions in the source document — typically expressed as bounding box coordinates or page-level coordinates. Technically, this requires the model to not only "understand" document content but also simultaneously output the spatial origin of each piece of information, essentially appending a precise localization task on top of the comprehension task. In RAG (Retrieval-Augmented Generation) pipelines, attribution information can also be used to verify whether a model's response is grounded in the source, helping prevent hallucinations. In compliance-sensitive industries (such as insurance claims or medical records), bounding box coordinates can directly drive UI highlighting so reviewers can locate source evidence in seconds — dramatically improving human-AI collaboration efficiency.

Accuracy Matters, But So Do Speed and Cost

Beyond accuracy, LlamaIndex also emphasizes the practical engineering constraints of real-world deployment: form parsing can't optimize for precision alone — it must also be viable in terms of cost and speed.

Enterprises process forms in high volumes and at high frequency. If a solution requires invoking a heavy model for every document, with long processing times and high per-call costs, then no matter how accurate it is, it simply can't scale. An ideal form parser therefore needs to strike a balance between accuracy, latency, and per-unit cost.

These four requirements — structured types, high accuracy, grounding and attribution, and low cost at speed — collectively define what the team calls "what it actually takes to build a form parser."

On the engineering side, the "cost" of form parsing typically spans two dimensions: inference latency (processing time per document) and monetary cost (per-thousand API call fees or self-hosted compute consumption). High-volume form processing scenarios — such as an insurance company processing tens of thousands of claims per day — are highly sensitive to both. A common engineering strategy is the "cascading model" approach: use a lightweight, fast model to handle simple forms, then route difficult cases to a heavier, more accurate model, balancing overall throughput against accuracy. LlamaIndex's explicit emphasis on cost and speed constraints also reflects a shared pain point in productizing document AI: solutions that excel on academic benchmarks often stall at the PoC stage because inference costs are simply too high.

Want to Try It?

LlamaIndex has integrated this capability into its document parsing product, LlamaParse, where developers can directly try the form model. The team also published an in-depth technical blog post walking through the full process of building a form parser — interested readers can dig into the original for more detail.

Broadly speaking, specialized document parsing models are becoming an important component of RAG and document intelligence pipelines. No matter how capable a general-purpose VLM is, it will struggle to cover verticals like form parsing that demand extreme structural precision. Purpose-built parsing solutions will continue to have clear value for the foreseeable future.