One in Three AI Medical Records Has Errors: Omission Is the Deadliest Failure Mode

When the Problem Isn't What AI Wrote — It's What It Left Out

An audit of three deployed AI medical record tools (AI scribes) has revealed a sobering finding: one in every three clinical notes contains a verifiable failure. More critically, the dominant failure type isn't inaccurate documentation — it's omission. Information that was clearly established during the clinical encounter simply never made it into the record.

This finding carries broad implications for any scenario that relies on LLMs to generate structured content. We tend to focus on what models "get wrong," but we routinely overlook a more insidious and dangerous problem: what models "leave out." In a medical context, a missing medication history, allergy, or symptom description can have consequences far more severe than a poorly worded sentence.

Why LLM Judges Are Near Coin-Flip at Detecting Omissions

The industry's standard response to AI-generated content quality issues is to deploy an "LLM judge" — have another large model read the generated note and cross-reference it against the original clinical transcript. This approach may work reasonably well for catching factual errors, but the audit found that it performs near coin-flip when it comes to detecting omissions.

The root cause is a structural flaw in how these judge mechanisms work: judges verify presence, not absence. When a judge reads a sentence in a clinical note, it can check whether that sentence is grounded in the transcript. But when a piece of information never appears in the note at all, there's no explicit "anchor" to trigger a check. The judge has no way to spontaneously realize, "something should have been here, but it's missing."

This reveals a deep cognitive bias: verifying what exists is natural; noticing what doesn't exist requires active, exhaustive retrieval. This is true for humans and models alike. A judge that simply "reads through and flags problems" is structurally incapable of reliably detecting silenced information.

This phenomenon has roots in cognitive science. Research on "change blindness" and "inattentional blindness" consistently shows that both humans and systems are far weaker at perceiving things that disappear than things that appear. LLMs, trained on co-occurrence patterns in language, have attention mechanisms fundamentally activated by tokens that are present in the input — absent tokens produce no activation signal and therefore trigger no reasoning process. This means the blind spot persists regardless of model scale, as long as the task structure remains unchanged. Notably, "near coin-flip" doesn't mean random distribution — it skews systematically toward false negatives: judges are more likely to incorrectly conclude that a missing item is present than to flag a present item as missing. This causes omission errors to be statistically and persistently underestimated.

The Real Fix: Restructure the Task, Not the Judge

The intuitive response to this problem is to "train a stronger judge." But the audit's conclusion points in the opposite direction: the problem isn't the judge — it's the task structure.

The proposed solution is to fundamentally restructure the detection task:

Step 1: Enumerate the Facts

Instead of asking a model to "read the note and find problems," start by exhaustively listing every fact from the original transcript that should have been documented. This step makes implicit, conversationally distributed information explicit and structured — producing a concrete checklist.

Technically, this step corresponds to an "Information Extraction" task, not a "quality assessment" task. The two make very different demands on the model: quality assessment requires the model to simultaneously hold an "expected state" and an "actual state" in mind and compare them — a cognitively expensive operation. Information extraction simply requires the model to identify and structure factual items from a single source (the transcript), which is a capability heavily reinforced during supervised fine-tuning and instruction-following training. In practice, this step typically requires pre-defining a "fact category" template (e.g., chief complaint, history of present illness, medication history, allergies, vitals, diagnosis, orders, etc.) to guide the model toward exhaustive category-by-category extraction rather than free generation — reducing category-level omissions. The structured output also serves as a basis for downstream auditing and traceability.

Step 2: Verify Item by Item

With this fact checklist in hand, check each item individually to see whether it is reflected in the generated note. Omissions are no longer "an imperceptible absence" — they become explicitly unchecked items on a list.

This "enumerate first, then verify" paradigm transforms the hard problem of "detecting absence" into the model-friendly problem of "detecting whether each item is present." It doesn't fight against the model's capability limits — it routes around them. Through task design, it makes absence visible.

Implications for AI Application Design

The value of this case extends far beyond medical records. It points to a universal engineering principle: when a class of errors is systematically invisible within an existing evaluation framework, the answer is usually to redesign the task — not to deploy a more powerful model.

Whether the application is document summarization, meeting minutes, legal contract review, or data extraction — any time an output needs to "completely cover" critical information from an input, omission is a natural failure mode. And any verification approach that simply "reads through the output and looks for mistakes" will break down when it comes to detecting omissions.

Structuring the verification process — first building a complete checklist of what should be present, then comparing item by item — is a pragmatic path toward more reliable AI outputs. It acknowledges the model's innate weakness in "perceiving absence" and compensates through task design.

Absence is the true failure mode. And the only way to fight absence is to explicitly name everything that should have been there.

There's a useful analogy from software engineering: test coverage. The value of unit testing isn't in "running the program and seeing if it breaks" — it's in predefining every behavioral assertion that must be satisfied, then verifying each one. AI output quality assurance needs to make the same conceptual shift — from "post-hoc inspection" to "spec-first verification." This also echoes formal verification: define a complete specification first, then prove the implementation satisfies it, rather than hoping random testing surfaces the bugs. For high-stakes AI applications — healthcare, legal, financial compliance — this "spec-first" verification paradigm may not only be an engineering best practice, but a baseline requirement for regulatory compliance.