The Pitfall of LLM Structured Extraction: Citations Are Correct, but Values Are in the Wrong Place

A Seemingly Perfect LLM Extraction Result

A developer built a PHP package using a local LLM (gemma4:e2b, running via Ollama on a Mac) to parse air-conditioning electricity bills. The pipeline design was standard: give the model a prompt, the linearized text of the bill PDF, and a JSON Schema (passed via the format parameter), and require the model to provide a citation for each row of data—a line copied verbatim from the original bill text, serving as proof of the source of the extracted values.

With the first version of the prompt, the model got almost all values right, but the citations only contained labels (e.g., "ACS") rather than the full line. The developer adjusted the prompt to specify: "The citation is the complete row of the table, exactly as printed, and the unit is the price unit (€/L), not the quantity unit." The result was encouraging: all 24 values were correct (6 fields × 4 rows), and all 4 citations were correct. It looked like a job well done.

Disaster Strikes: All Citations Correct, but Values Misaligned

The problem lay in a subsequent "whitespace normalization" step. After the developer normalized the extracted text and fed it to the LLM, the value score plummeted from 24/24 to 6/24—while the citations remained 4/4 correct.

All values existed in the document, but they were placed in the wrong fields. Take the ACS row as an example:

Field	Expected Value	Model Output
quantity	2.475,00	2.475,00 ✓
price	0,00740931	0,00740931 ✓
vat_percent	21,00	0.00000000 ✗
base	18,34	21,00 ✗
taxes	3,85	18,34 ✗
subtotal	22,19	3,85 ✗

Every amount shifted one column to the right. The citations proved that "these values do exist on the bill," but they could not prove that "each value was placed in the correct field."

The "citation" mechanism in LLM structured extraction refers to requiring the model, when outputting each extracted field, to also provide the verbatim source snippet from the original document for that value. The theoretical basis for this approach is: if the model can accurately copy the source text, it demonstrates that it actually "saw" the value rather than fabricating it, thereby reducing the risk of hallucination. Citation validation typically only needs to check whether the citation snippet exists in the original text—it is cheap to implement and highly interpretable, so it is widely used as a quality gate in document parsing pipelines. However, the essence of citation validation is existence checking—it only answers "does this value appear somewhere in the document," not "does this value come from the correct row-column intersection." When the document is flat text and has lost column-separator information, multiple values may exist on the same line simultaneously; the citation snippet can still match successfully, but the field mapping is already misaligned. This is the root cause of why citation validation easily creates a false sense of security.

Root Cause: Tabs Were Erased by Normalization

The raw text output by the PDF parser used tabs to separate columns:

Concepto → Cant. → Tarifa → %IVA → Base → Impuestos → Subtotal
ACS → 2.475,00 → 0,00740931 €/L → 21,00 → 18,34 € → 3,85 € → 22,19 €

The arrows represent tabs; the 7-column header corresponds to 7 data cells. The model only needed to count tabs to locate column boundaries. But after the developer's normalization converted all characters to spaces:

Concepto Cant. Tarifa %IVA Base Impuestos Subtotal
ACS 2.475,00 0,00740931 €/L 21,00 18,34 € 3,85 € 22,19 €

The data row expanded from 7 elements to 11, and the model lost a reliable basis for determining cell boundaries—it likely made counting errors, causing each value to shift one column.

Tabs and column boundaries in PDF text extraction: PDFs themselves do not store "table" structure—they only store the coordinate positions of characters. Parsers typically infer column separators based on character spacing and approximate them using tabs (\t) or consecutive spaces. For multi-column tables such as bills and invoices, tabs are the only low-cost clue for the model to determine "which cell corresponds to which field." Once a normalization step (e.g., unifying whitespace or removing extra spaces) replaces tabs with single spaces, a data row that originally had 7 columns will expand into more tokens because the field values themselves contain spaces (e.g., "0,00740931 €/L"), and the model can no longer reliably count column boundaries. More insidiously, this damage occurs in the preprocessing stage—the prompt and Schema remain unchanged, citation validation still passes, and therefore the problem is not exposed in the logs. Preserving the original tab structure of the model input, or retaining column-separator markers during normalization, is key to avoiding such misalignment.

Three Layers of Validation: Each Has Strengths and Weaknesses

This case reveals the blind spots of three common validation methods in LLM structured extraction:

Check	What It Can Prove	What It Misses
JSON Schema	Fields exist, types are correct	Values placed in the wrong fields
Citation	Values exist in the corresponding row of the document	Values located in the wrong column
Deterministic cross-calculation	Values are mathematically consistent with each other	Errors that preserve calculation correctness (rare but possible)

Schema guarantees structure, and citations guarantee existence, but neither can guarantee positional correctness. Citation validation is especially prone to creating a false sense of security—a 4/4 citation pass rate masked 18 value errors.

The JSON Schema format parameter and structured output: In local inference frameworks such as Ollama, the format parameter allows passing a JSON Schema to constrain model output to conform to specified field names, types, and nested structures. This effectively prevents the model from improvising, omitting fields, or returning unparseable text, and is a standard feature of modern LLM extraction pipelines. But Schema constraints operate at the syntax level: they only check whether vat_percent is a number and whether base exists—not whether the number in base came from the bill's Base column. Therefore, when the model puts 21.00 into base and 18.34 into taxes, Schema cannot detect it at all—both fields are valid floats. The combination of Schema and citation validation covers the two dimensions of "structurally correct" and "value exists," but conveniently leaves the blind spot of "positionally correct." Understanding the proof boundary of each validation method is more important than stacking more checks.

Hidden Risks in Production

The developer was able to discover this problem because there was a local test file with known correct answers. But in production, such a reference does not exist. The same misalignment can occur silently when a supplier changes the bill layout, OCR is introduced, or the PDF extractor is replaced—the input changes, the structure disappears, and citations still all pass.

This is a more insidious problem than "hallucination": the model did not fabricate nonexistent values (citation validation passed), yet it systematically placed them in the wrong positions. Pipelines that rely solely on citations as a quality gate will let erroneous data through directly.

Three Actionable Improvement Principles

1. Do not alter the model input. Keep the text sent to the LLM in its original tab-delimited format. Normalization should only be used for comparison, and only on a copy.
2. Citations are not sufficient evidence. They can only prove that the value exists in the document. You need deterministic checks that connect relationships between values: quantity × price = base, sum of rows = total.
3. Measure before you trust. Every time you change the prompt, model, or PDF extractor, rerun scoring against bills with known ground-truth values. Without this scoring mechanism, a 4/4 citation pass rate will make you mistakenly believe everything is fine.

Cross-calculation shows unique value here: for the ACS row, quantity × price = 2475 × 0.00740931 = 18.34, but the model put 21.00 into the base field—the calculation fails and the extraction is rejected. This deterministic check does not rely on the model proving itself, and is an indispensable third line of defense beyond citations and Schema.

Deterministic cross-calculation as the third line of defense: Cross-calculation refers to using known mathematical relationships in the domain (e.g., quantity × unit price = base, base + taxes = subtotal, sum of row subtotals = total) to perform consistency checks on extraction results. Its advantage is that it does not rely on the model proving itself, nor does it rely on format information from the original document—as long as the numerical relationships hold, it can reverse-confirm whether the field mapping is correct. In the above case, if the model puts 21.00 into base, then quantity × price = 18.34 ≠ 21.00, and validation immediately fails, rejecting that row. The limitation of cross-calculation is that it can only cover fields with mathematical relationships, and it cannot do anything about "errors that preserve calculation correctness" (e.g., an entire row of values scaled uniformly)—but it is extremely effective against systematic errors such as column misalignment. In practice, you should prioritize identifying reliable identities in bills and use them as hard gates after extraction, rather than relying only on citation pass rates.

A Takeaway for Everyone Doing LLM Extraction

When your pipeline shows "citation validation 4/4 passed," it is worth asking: what exactly do citations prove? They prove the value is in the document; they do not prove the value is in the correct field. The reliability of structured extraction ultimately depends on how you validate the mapping between fields and values, not on whether the values exist.