Intelligent Document Processing (IDP) Scoring Methodology: Confidence Is Not Correctness

Why IDP Needs a Dedicated Scoring Method

The core task of Intelligent Document Processing (IDP) is to extract key information from structured documents (especially forms)—known as Key Information Extraction (KIE). However, many teams directly apply OCR metrics or simple accuracy when evaluating extraction quality, and the results are often distorted. A researcher on Reddit published a two-part Medium series that systematically lays out practical methods for IDP scoring, emphasizing an easily overlooked principle: confidence is not correctness.

Rather than proposing new algorithms, the series offers an "investigative" summary of current practice, covering the full chain from raw result recording to confusion matrix mapping and then to metric selection. The author's focus is on "how to measure," not "how to implement," which is especially helpful for interpreting the results of different benchmarks.

KIE (Key Information Extraction) is the core subtask within IDP. Its goal is to extract predefined key fields (such as amount, date, and vendor name) from semi-structured or structured documents (e.g., invoices, forms, contracts) and output them as structured data. Unlike general OCR, which only cares about "whether characters are recognized correctly," KIE requires the model to understand the semantic boundaries of fields and document layout, so its evaluation must be based on field-level exact matching or semantic equivalence judgments rather than character-level error rates. The difficulty of KIE lies in the fact that the same information may appear in different formats (e.g., the date "2024-01-01" vs. "01/01/2024"), that determining blank fields depends on context, and that the model may "fabricate" content that does not exist (hallucination). These characteristics mean KIE scoring requires dedicated confusion matrix mapping rules and metric selection and cannot directly reuse OCR's CER/WER.

From Raw Results to Confusion Matrix: Record First, Then Map

The first step in scoring is not to calculate metrics directly, but to faithfully record raw results before mapping them to a confusion matrix. The author suggests classifying each field's extraction result into one of five raw states: correct, misread, missed, invented, and correct blank. This recording approach preserves unfiltered information and avoids bias in later metric selection.

The next step is mapping to the four values of the confusion matrix (TP, FP, TN, FN). But the mapping rules themselves need explicit agreement, especially for text fields and checkboxes. The article discusses in detail different conventions for handling misreads: counting only as FN, only as FP, or as both FN and FP. Different conventions directly change the final score, so the convention used must be stated when publishing results.

Blanks, NULLs, and Unreadable Values: The Gray Areas of Scoring

In form extraction, how to handle blank fields, NULL values, unreported checkboxes, and unreadable values is where scoring disagreements most easily arise. The author lists several counting approaches and analyzes their impact on metrics. For example, should correctly identified blank fields (correct blank) count as True Negatives? If so, Accuracy is significantly inflated.

The article points out that counting "correctly identified blanks" this way makes Accuracy look higher but may mask the model's shortcomings in extracting real information. Therefore, when choosing a metric, you must understand the counting logic behind it rather than looking at a single number.

F1, Recall, and Accuracy: Choosing the Right Metric Matters More Than Tuning

The article compares the applicability of F1, Recall, and Accuracy in IDP scenarios. A key insight is that Accuracy tends to be inflated when there are many blank fields, because correctly identified blanks are counted as correct. The Jaccard Index (IoU) is described as "Accuracy without True Negatives" and can be converted to F1. The author also discusses the role of Precision, Specificity, and False Positive Rate in controlling LLM hallucination—when the model "fabricates" content, FP rises and Precision falls, so these metrics are more sensitive to hallucination monitoring.

In addition, the article covers the differences between micro, macro, and weighted F1, and how to choose a "headline score" while clearly stating the conventions behind it. The author does not recommend one "correct" approach but presents different variants and their trade-offs.

Jaccard Index (Intersection over Union, IoU) is often used in IDP scoring to measure the overlap between the set of extracted fields and the set of ground-truth fields, calculated as the intersection size divided by the union size. The article describes it as "Accuracy without True Negatives," because Jaccard only focuses on the overlap of positive examples (extracted fields) and does not involve counting correctly identified blanks (TN). This characteristic makes it more robust than Accuracy in form scenarios with many blank fields. There is a mathematical relationship between Jaccard and F1: when the Precision and Recall of positive examples are determined, Jaccard = F1 / (2 - F1), so they can be converted into each other. Understanding this relationship helps teams maintain consistency when reporting a "headline score" and avoid misjudging model quality due to different metric choices.

Confidence Calibration and KIE vs. OCR Metrics

The article specifically discusses the separation of confidence and correctness, and introduces calibration metrics ECE (Expected Calibration Error) and MCE (Maximum Calibration Error) to measure whether model confidence is reliable. A high-confidence wrong prediction is more dangerous than a low-confidence error because it misleads downstream decisions.

Finally, the author distinguishes KIE metrics from OCR metrics: OCR commonly uses CER (Character Error Rate) and WER (Word Error Rate), but these metrics focus on transcription accuracy rather than the semantic correctness of information extraction. IDP scoring needs KIE-level metrics, because a single character error can make a field value completely wrong, and OCR metrics cannot capture this semantic-level failure.

ECE (Expected Calibration Error) and MCE (Maximum Calibration Error) are standard metrics for measuring the reliability of model confidence. ECE buckets prediction confidence (e.g., 0-10%, 10-20%, ...) and computes the weighted average of the difference between the average confidence and the true accuracy in each bucket; MCE takes the maximum deviation across all buckets, reflecting the worst-case calibration error. In IDP scenarios, a high-confidence wrong prediction (e.g., the model outputs an incorrect amount with 95% confidence) directly misleads downstream automation processes, so ECE/MCE reveal the risk of "confidently making mistakes" better than plain accuracy. A well-calibrated model should have confidence matching correctness, and teams can use this to set confidence thresholds and route low-confidence results to human review.

Practical Recommendations: Transparent Conventions, Report Multiple Metrics

Taken together, the series shows that there is no single "correct" answer for IDP scoring, but there are several points of consensus:

• Fully record raw result states before mapping to a confusion matrix;
• Clearly state the counting rules for blanks, NULLs, checkboxes, and unreadable values;
• Do not report only one metric; at minimum provide Precision, Recall, and F1 together and state the conventions;
• For extraction tasks involving LLMs, pay attention to the False Positive Rate to control hallucination;
• Use ECE/MCE to check whether confidence is reliable and avoid "confidently making mistakes."

Although this methodology is not breakthrough research, it provides an actionable scoring framework for teams that need to compare different IDP solutions or interpret benchmark results. The author emphasizes not endorsing any vendor, and the series focuses on scoring itself—this neutrality makes the discussion more valuable as a reference.

One-line Summary

IDP scoring requires recording raw results first, then mapping to a confusion matrix, choosing the right metrics, and calibrating confidence.

Paragraph Summary

This article systematically reviews the scoring methodology for Key Information Extraction (KIE) in Intelligent Document Processing (IDP), emphasizing that confidence is not correctness. It points out that the first step in scoring is to faithfully record five raw result states (correct, misread, missed, invented, correct blank), then map them to a confusion matrix, and the mapping conventions (e.g., whether a misread counts as FN or FP) must be publicly stated. How blank fields and checkboxes are counted significantly affects Accuracy, and correctly identified blanks can inflate accuracy. The article compares the applicability of F1, Recall, and Accuracy, noting that the Jaccard Index can be viewed as Accuracy without TN and can be converted to F1, while Precision and False Positive Rate are more sensitive to LLM hallucination. Finally, it introduces ECE/MCE calibration metrics and distinguishes KIE metrics from OCR's CER/WER. Practical recommendations include transparent conventions, reporting multiple metrics together, using FP rate to control hallucination, and using calibration to check confidence reliability.