Silent Cue Loss in Enterprise Document Ingestion: Why Agents Confidently Get It Wrong

When enterprise agents execute tasks, the first step is often not "thinking" but "reading documents." Whether following policy, answering compliance questions, or submitting tickets, they must first convert Word or PDF files into tokens. The problem lies precisely at this layer: mainstream document converters systematically strip away semantic cues that authors encode through formatting—information such as highlights and colors is removed during conversion, causing agents not to be "visibly incomplete" but "confidently wrong."

The Layer of Information Eaten by Format Converters

Researchers built a task based on 76 real policy documents: agents were asked to identify requirement items that authors had deliberately marked, using the files' own OOXML properties as the deterministic matching criterion, with no LLM acting as judge at any point. The results show that when the ingestion layer "cannot see" formatting, performance plummets from a micro F1 of 0.690 to 0.150 (docling) and 0.075 (plain text). This gap reproduces across six answer models—five models spanning Claude's Opus, Sonnet, and Haiku tiers, plus OpenAI's GPT-5.6.

More alarming is the failure mode. Agents that cannot see the cues simply "guess": docling's false positive rate on requirements not marked by the author is four times ours. In other words, format loss doesn't make the agent say "I don't know"—it makes it confidently give the wrong answer.

micro F1 is a common evaluation metric in information extraction and classification tasks. It first aggregates predictions across all classes (here, "author-marked requirement items" and "non-requirement items"), computes global precision and recall, then takes their harmonic mean. Unlike macro F1, which averages each class with equal weight, micro F1 is dominated by classes with large sample sizes, making it better suited to measuring overall extraction quality. In this study, the sharp drop in micro F1 means the agent as a whole both missed many true requirement items and misjudged many non-requirement items—not a local fluctuation in any single class.

False positive rate refers to the proportion of content the model misjudges as requirement items that the author did not mark as such. The article notes that docling's false positive rate is "four times" that of the format-preserving approach, indicating that format loss not only causes missed detections but also significantly increases over-identification—the agent treats ordinary paragraphs as obligatory clauses, which is the quantitative embodiment of "confidently giving the wrong answer."

Highlights and Colors Are the Key Cues

Span-level leave-one-out analysis localizes the loss to two cues: highlight (+0.302) and color (+0.262), both with Holm-corrected p-values below 1e-4. By contrast, bold, italic, monospace, and strikethrough all have effects within ±0.003—essentially zero. The same asymmetry appears in PDFs.

Color is particularly special: it is a cue that mainstream converters generally do not preserve, and a dimension that existing parsing benchmarks never score, yet it is quite common in real documents—32.8% of 6,460 documents contain deliberate color or highlighting, with 10.3% applied to obligatory clauses. This means critical information in a large number of enterprise documents sits in a blind spot where "conversion means loss."

Span-level leave-one-out analysis is an ablation experiment design: formatting cues in a document are removed one span (text fragment) at a time, and the drop in model performance after removing a given type of cue is observed to attribute each cue's marginal contribution. The "+0.302" and "+0.262" here are the drops in micro F1 after removing highlights or color—the larger the value, the more critical the cue.

Holm correction is a classic method for controlling false positives in multiple comparisons. The study simultaneously tests multiple cues—highlight, color, bold, italic, monospace, strikethrough—and without correction, the probability of accidental significance rises with the number of tests. Holm correction adjusts thresholds stepwise from smallest to largest p-value, ensuring the overall probability of a Type I error remains controlled; the p<1e-4 reported here remains significant after correction, indicating that the effects of highlight and color are not artifacts of multiple testing.

Low Fix Cost, and Measurable

A pre-registered experiment shows that explicitly "typing a cue" for the agent yields an additional +0.085 improvement (Holm-corrected p=0.049) beyond merely preserving the cue. The research team also validated label quality through an independent modality—rendered page images (kappa=0.72)—and established the upper bound achievable by wording alone (AUC 0.664).

For agent builders, the conclusion is straightforward: the decisive factor in this class of reliability problems lies in the ingestion layer, not the reasoning layer. The fix is cheap and quantifiable—preserving color and highlights and actively annotating these cues explicitly for the model can significantly close the gap. From a multi-source cross-validation perspective, the study's task design, model reproduction, and statistical testing corroborate one another, giving the core conclusions high credibility.

Pre-registered experiments mean publicly registering the hypothesis, sample size, primary metrics, and analysis plan before the study begins, then executing according to the registered plan afterward. This effectively curbs p-hacking behaviors such as cherry-picking favorable results after the fact or arbitrarily adjusting analysis criteria. Thus the phrasing "pre-registered" plus "Holm-corrected p=0.049" means the +0.085 improvement barely reached significance under a pre-agreed test, giving it higher credibility than post-hoc analysis.

Cohen's kappa (kappa=0.72) is a statistic measuring agreement between two annotators, with chance agreement subtracted out. Generally, 0.61–0.80 is considered "substantial agreement"; 0.72 indicates that label quality independently validated via rendered page images is reliable and not the researchers' subjective assumption. AUC 0.664 measures the ability to distinguish requirement items from non-requirement items using wording text alone. 0.5 is equivalent to random guessing; 0.664 is relatively weak discrimination, used to define the performance ceiling achievable "without relying on formatting, looking only at text."

Implications for Enterprise Deployment

Most teams focus on prompts and model selection while assuming document parsing is "solved" infrastructure. This study reminds us that silent information loss at the parsing layer translates directly into business risk: compliance judgments, policy enforcement, and ticket classification can all go astray because of a single dropped highlight. We recommend adding acceptance tests for format cue preservation to document ingestion pipelines and making "cue awareness" a standard dimension in agent evaluation.

One-line Summary

Document conversion loses highlight and color cues, causing enterprise agents to drop from 0.690 to 0.150 and confidently make mistakes.

Paragraph Summary

The article points out that the first step for enterprise agents executing tasks is document ingestion, yet mainstream converters systematically strip semantic cues authors encode through formatting. Experiments based on 76 real policy documents show that format-blind ingestion causes micro F1 to plummet from 0.690 to 0.150 (docling) and 0.075 (plain text), reproducing across six models. Span-level leave-one-out analysis localizes the loss to highlight (+0.302) and color (+0.262), while bold, italic, and similar cues have near-zero effect. About 32.8% of enterprise documents contain deliberate color or highlighting, with 10.3% applied to obligatory clauses, yet converters do not preserve them and benchmarks do not score them. Pre-registered experiments show that explicitly typing cues yields an additional +0.085 improvement; the fix is cheap and measurable, and the decisive factor lies in the ingestion layer rather than the reasoning layer.