Why Do Coding Agents Get Stuck on the Last Mile of Long-Document Conversion? An Empirical Study on Clinical Trial Protocol Digitization

The Real Dilemma of Long-Document Digitization: Finding the Evidence but Losing on Transcription

Clinical trial protocols are the "source of truth" for study design, yet the vast majority still exist as unstructured PDFs. Converting these protocols into structured, machine-readable data enables clinical software to analyze, validate, and even execute them. Coding agents—AI systems capable of autonomously selecting tools, executing code, and revising strategies based on results—are increasingly being applied to such long-document tasks.

But a key question remains unanswered: to what extent can they reliably convert complete documents into accurate structured output?

An empirical study on this question offers a sobering answer. Using clinical trial protocol digitization as the test scenario, researchers built TrialPD-Bench—a benchmark set with expert annotations, comprising 150 public protocol PDFs and covering four task types. Evaluation extended beyond final answer correctness to include evidence recall in agent trajectories, agent behavior patterns, and cost.

What Are Unstructured PDFs and Structured Data

An unstructured PDF is a file in which content is presented only through visual layout (text position, font size, table lines) without semantic tags. The human eye can understand which visit and which column "systolic blood pressure ≤ 140 mmHg" belongs to, but a machine sees only coordinates and character streams. Structured data requires the same information to be decomposed into key-value pairs with field names, such as {visit: "Week 4", criterion: "SBP", operator: "<=", value: 140, unit: "mmHg"}, so that it can be directly queried, compared, and validated by programs.

The conversion between the two is not simple OCR: it requires the model to simultaneously perform layout understanding (table boundaries, cross-page continued tables, footnote attribution), semantic disambiguation (the same abbreviation meaning different things in different sections), and normalization (unifying units, date formats, and enumerated values). Clinical trial protocols are especially demanding—a single transcription deviation in a value or unit could change the outcome of an inclusion/exclusion criterion determination, which is why this task is regarded as a "stress test" for long-document extraction.

What Is an Agent Trajectory

An agent trajectory refers to the complete sequence of actions left by an agent during a single task execution: which tools it invoked, which pages or fragments of the document it read, what intermediate code it generated, what results it received, and how it adjusted its next step accordingly. Unlike "black-box" evaluation that only records inputs and outputs, it exposes the reasoning and operation process.

For long-document tasks, the trajectory is key evidence for locating the failure point. By computing evidence recall—the proportion of source-text fragments on which the final answer depends that were actually read in the trajectory—one can distinguish two fundamentally different types of failure: one is "didn't find it" (retrieval failure, where relevant passages were never accessed), and the other is "found it but wrote it wrong" (transcription failure, where evidence appears in the trajectory but is not correctly carried into the output structure). The core diagnosis of this study is based precisely on this distinction, which is how it concludes that the bottleneck lies in the "last mile" rather than the retrieval stage.

Core Finding: Stuck on the "Last Mile"

The research team tested nine proprietary and open-weight models across five coding agent harnesses. The conclusion was fairly direct: even the strongest agents did not fully solve the task of protocol digitization.

A more specific diagnosis points to a clear bottleneck—agents can usually find the evidence they need, but fail on the "last mile" of transcription and structuring. In other words, the problem is not "not knowing where to look," but "being unable to accurately carry it over once found."

This finding offers a finer characterization of the capability boundaries of current coding agents: retrieval and localization capabilities have reached a relatively high level, but precise information extraction and format conversion across paragraphs and tables in long-document scenarios remain weak links.

Filling the Gap with a Domain-Specific Framework: TrialScribe's Verification-Loop Approach

To address the "last mile" problem, the researchers developed TrialScribe—a domain-specific agent framework that equips coding agents with task skills and a verification loop.

Experimental results: TrialScribe improved locally deployed open-weight models by 6.2 to 59.4 points across eight metrics, enabling them to match or even surpass proprietary models such as GPT-5.6-Sol.

The significance of this improvement magnitude deserves unpacking. It suggests that the limited performance of open-weight models under general-purpose coding agent frameworks may not stem from insufficient foundational capability, but rather from a lack of domain knowledge and self-verification mechanisms. When "skills" and "verification" are added at the framework level, the capability gap can be substantially bridged, or even reversed.

Task Skills and Verification Loop

Task skills refer to explicitly encapsulating domain knowledge into reusable operational units for the agent to invoke when needed. For example, skills such as "identify inclusion/exclusion criteria tables," "normalize dosing frequency abbreviations," and "compare cross-page continued tables" essentially crystallize experts' tacit experience into tools or prompt templates with defined input-output contracts, reducing the burden on the model to infer from scratch each time.

The verification loop adds a self-check after generating structured results: looking up output fields against the source text and confirming item by item whether values, units, and qualifiers are consistent with the source, triggering re-extraction or correction if not. Its difference from simply "asking the model again whether it's right" is that the basis for verification is the source text rather than the model's own judgment, so it can catch confident but erroneous transcriptions. The combination of the two forms a closed loop of "specialize first, then review," which is precisely why open-weight models achieve a substantial leap after this link is filled—the foundational model capability is unchanged; what changes is the domain constraints and error-correction opportunities provided at the framework level.

Implications for Long-Document AI Applications

This study has reference value for a broader range of AI application scenarios. Tasks such as contract review, financial report parsing, and regulatory comparison are essentially isomorphic to protocol digitization: all involve extracting structured information from unstructured long documents.

The signal the study conveys is that merely increasing model scale or general coding capability may not break through the "last mile." More effective directions include:

• Evidence traceability: Evaluation should not look only at final answers but examine evidence recall in the trajectory, which can locate which link the failure occurred in.
• Domain frameworks first: Building skill libraries and verification loops for specific document types yields significant returns.
• Verification closed loop: Having agents self-check and look up the source text after transcription is a key mechanism for remedying structuring errors.

For teams currently deploying coding agents into production for long-document processing, the benchmark, framework, and diagnostic methods provided by this study constitute an evaluation and improvement path that can be directly referenced.

Frequently Asked Questions

Can coding agents fully replace humans in digitizing clinical trial protocols?

Not currently. The study shows that even the strongest agents have not fully solved the task, with the bottleneck concentrated in transcription and structuring accuracy.

What dimensions does TrialPD-Bench evaluate?

In addition to final answer performance, it also includes evidence recall in agent trajectories, agent behavior, and cost.

How much does TrialScribe improve open-weight models?

It improves them by 6.2 to 59.4 points across eight metrics, matching or surpassing proprietary models such as GPT-5.6-Sol.

One-line Summary

The study finds that coding agents can find evidence in long documents but get stuck on the last mile of transcription and structuring.

Paragraph Summary

Using clinical trial protocol digitization as its scenario, the study built TrialPD-Bench, a benchmark containing 150 expert-annotated PDFs and covering four task types, and evaluated five agent frameworks and nine proprietary and open-weight models. The results show that even the strongest agents cannot fully solve the task, and the bottleneck is not retrieval but the "last mile" of transcription and structuring: evidence is often found but cannot be accurately carried into structured output. To address this, the researchers proposed the domain-specific framework TrialScribe, which through task skills and a verification loop improves local open-weight models by 6.2 to 59.4 points across eight metrics, matching or even surpassing proprietary models such as GPT-5.6-Sol. The study also emphasizes that evaluation should incorporate trajectory-level evidence recall and offers an improvement path transferable to long-document tasks such as contract review and financial report parsing.