From Spot Fixes to Measured Gains: Optimizing Document Extraction with ai_extract and MLflow

Document extraction is the first hurdle in many AI applications, but its optimization process is often frustrating: you find a field extracted incorrectly, adjust the prompt or rules, and then hope the next document won't reveal a new problem. This "spot fixes" approach seems agile, but in reality it traps teams in an endless firefighting loop, unable to answer a fundamental question—how much has our extraction accuracy actually improved?

This article, drawing on practical experience with ai_extract and MLflow Prompt Registry, explores how to transform document extraction optimization from scattered, experience-driven fixes into a measurable, reproducible engineering process.

Three Structural Flaws of the Spot-Fix Mode

Spot fixes are popular because they provide immediate feedback at low psychological cost. But their price only becomes apparent when projects scale.

No baseline, no attribution. After tweaking a prompt, you feel it's "probably more accurate," but you can't rule out that the test samples just happened to be easier. Without a fixed evaluation set, every change is blind.

Scattered knowledge, hard to reuse. The experience of fixing a particular invoice field might exist only in an engineer's chat history. The next similar project starts from scratch and repeats the same mistakes.

Invisible regression risk. A prompt that fixes field A might silently break the extraction of field B. Without automated evaluation, such degradation goes unnoticed until online users complain.

The root cause of these flaws is not insufficient model capability, but the lack of a "measurement" step in the engineering process.

Gold Standard Dataset refers to a collection of documents manually annotated and used as an evaluation benchmark. In document extraction scenarios, it must cover the layout diversity, field variations, and edge cases (such as blurry scans, handwritten annotations, mixed languages) that may occur in real business. The quality of the gold standard directly determines the credibility of evaluation conclusions: if the annotations themselves are ambiguous or incorrect, all subsequent metrics will be distorted. Therefore, when building it, you should establish clear annotation guidelines, double-review controversial samples, and record inter-annotator agreement (e.g., Cohen's Kappa). Once the dataset is finalized, its version should be frozen, and it should only be expanded according to a defined process when coverage gaps are found, avoiding the self-deception of "switching test sets to boost metrics."

ai_extract: Decoupling Extraction Logic from Code

The core value of ai_extract lies in separating the "definition" of document extraction from its "execution." Developers describe the fields to be extracted and their types through a declarative schema, and the extraction logic itself is no longer hardcoded in business code.

This decoupling brings two direct benefits. First, changes to field definitions do not require modifying application code, reducing the probability of introducing bugs. Second, extraction configurations can be version-controlled like code, laying the foundation for subsequent measurement and comparison.

Once extraction logic becomes an independently manageable asset, the next step is naturally to incorporate it into a trackable experiment system—this is where MLflow Prompt Registry comes into play.

MLflow Prompt Registry: Making Every Prompt Change Traceable

Prompt Registry treats prompts as first-class engineering artifacts, recording their versions, change contents, and associated evaluation results. In document extraction scenarios, this means:

• Every prompt adjustment for a specific field generates a new version instead of overwriting the old one;
• Each version can be bound to corresponding evaluation metrics, so fluctuations in accuracy are documented;
• Teams can roll back to any historical version without relying on the verbal memory of "that prompt that worked last time."

By combining ai_extract's schema definitions with Prompt Registry's version management, the extraction process shifts from "change code—visual check—deploy" to a closed loop of "change configuration—automated evaluation—compare metrics—decide whether to deploy."

Prompt Versioning follows similar principles to code versioning, but targets natural language instructions as a special artifact. A prompt version should include not only the text itself but also its template variables, the model identifier used, inference parameters such as temperature, and the creation time and author. In MLflow Prompt Registry, each registration generates an immutable version number and supports an alias mechanism—for example, marking an evaluated and verified version as production and an experimental version as staging. Thus, online services reference prompts via aliases and can switch versions without code changes. Combined with evaluation metrics, teams can compare prompt versions just like comparing code commits, turning "which prompt is better" into reproducible experimental conclusions.

Building a Measurable Evaluation Loop

The key to moving from spot fixes to quantitative improvement is to build a stable evaluation process. Here are actionable steps:

1. Freeze the evaluation dataset. Collect a representative set of documents covering common layouts and edge cases, and manually annotate the correct results as the gold standard. This dataset should remain stable, occasionally expanded but not arbitrarily replaced.

2. Define field-level metrics. Overall accuracy masks local problems; only field-level accuracy can pinpoint specific weaknesses. Set higher passing thresholds for critical fields.

3. Run full evaluation on every change. Whether adjusting prompts, switching models, or modifying the schema, first run the evaluation set, record the metrics, and then decide whether to merge.

4. Replace subjective judgment with version comparison. The version history in Prompt Registry, combined with evaluation metrics, makes "did this change work?" a data question rather than a meeting debate.

Field-level Accuracy refers to the extraction correctness calculated separately for each independent field in the schema, rather than an overall accuracy that aggregates all fields. For example, an invoice may contain fields such as invoice number, amount, date, and tax ID; an overall accuracy of 90% could mask a serious problem where "the tax ID field is only 60% correct." Field-level metrics usually need to distinguish exact matches, fuzzy matches (e.g., date format differences), and semantic equivalence (e.g., amount unit conversion), and set different passing thresholds based on business importance. Critical fields (such as amount or contract number) should have stricter thresholds, and weighted scoring can be introduced. Field-level metrics also help identify error patterns: whether a certain layout causes multiple fields to fail simultaneously, or whether the prompt description for a particular field is unclear, thereby guiding more targeted optimization.

From Point Optimization to Systematic Improvement

Once the above loop is running, the team's behavior patterns change subtly. Engineers no longer ask, "Why did this field go wrong again?" but rather, "What was the accuracy for this field in the previous version, and which fields did this change affect?" Optimization shifts from passive response to proactive experimentation.

There is no silver bullet for improving document extraction accuracy, but there is a clear path: turn every fix into a measurable experiment, and distill the conclusions of each experiment into a traceable version. ai_extract makes extraction definitions clear and manageable, while MLflow Prompt Registry makes change history transparent and comparable. Combined, the quagmire of spot fixes can become a compounding cycle of continuous accumulation.