How LLM Agents Automatically Build CDEs and Extract Data from Sarcoma Case Report Abstracts

LLM agents induce CDEs from sarcoma case abstracts and extract evidence, with high relevance but insufficient correctness.
This study designed a two-stage LLM agent pipeline that proposed 72,909 candidate CDEs from 24,588 PubMed sarcoma case report abstracts, compressed them to 1,570 via agentic consolidation, and retained 53 CDEs supported by at least 50 abstracts for large-scale evidence extraction. All retained CDEs were successfully processed in 79.2% of abstracts. Manual review showed approval rates of 83.0% for relevance, 67.9% for clarity, and only 56.6% for correctness. Among 1,116 non-empty answer pairs from 100 sampled abstracts, exact match between LLM and manual extraction reached 94.9% (95% CI 93.0–96.4%), with errors mainly in anatomical and temporal fields. The authors conclude agents accelerate CDE drafting and extraction, but expert curation remains essential.
Clinical research data comparability has long been constrained by the manual definition process for Common Data Elements (CDEs). A recent study from PubMed attempted to use large language model agents to automatically induce CDE schemas from sarcoma case report abstracts and perform evidence-linked extraction on a large-scale corpus. This work demonstrates a practical pathway for agents in medical informatics while also exposing the boundaries of current methods.

From 24,588 abstracts to 1,570 CDEs: a two-stage agent workflow
The research team designed a two-stage agent pipeline. In the first stage, the agent proposed 72,909 candidate CDEs from 24,588 PubMed sarcoma case report abstracts; subsequently, through agentic consolidation, the candidates were compressed to 1,570 CDEs. The second stage focused on quality filtering, retaining 53 CDEs supported by at least 50 abstracts each, and attempting evidence-linked extraction across the entire corpus.
This "diverge then converge" strategy is noteworthy: the number of candidate CDEs differs from the final retained count by three orders of magnitude, indicating that the agent tends to over-generate during the induction phase, and that consolidation and frequency filtering are key steps for noise control. For clinical informatics teams, this means the agent does not replace manual definition but shifts human effort from "drafting from scratch" to "review and pruning."
Common Data Elements (CDEs) are standardized fields predefined in clinical research to unify data collection, such as "primary tumor site" or "diagnosis date," whose names, definitions, value formats, and coding must be explicitly specified. The purpose of CDEs is to enable data collected across different studies and institutions to be compared and aggregated. Traditionally, CDEs are manually developed by domain experts through consensus conferences, which is time-consuming, costly, and difficult to cover rare diseases or rapidly evolving subspecialties. This study attempts to automatically induce CDE schemas from literature using large language model agents, essentially exploring automation of the "data element discovery" step rather than replacing existing CDE standards themselves.
Extraction coverage and manual review: 83% relevant, but only 56.6% correct
In the large-scale extraction phase, all retained CDEs were successfully processed in 79.2% of abstracts, including cases returning "not mentioned" responses. This metric reflects the agent's robustness to long-tail expressions but also indicates that about one-fifth of abstracts had processing failures.
More critical are the manual review results. Two human reviewers' approval rates for retained CDEs were: relevance 83.0%, clarity 67.9%, correctness 56.6%. High relevance paired with low correctness suggests that the CDE directions proposed by the agent are generally on track, but specific definitions and boundaries still require expert correction. This gap aligns with the authors' conclusion: agent-assisted schema drafting should serve as a precursor to expert curation, not the endpoint.
Evidence-linked extraction refers to identifying and structuring specific values or evidence fragments related to a given CDE from text, for example extracting from "osteosarcoma of the distal left femur" the anatomical site "distal femur" and laterality "left." Unlike open-domain information extraction, the extraction targets here are constrained by pre-induced CDE schemas, making it a schema-driven extraction task. Evaluation typically distinguishes two levels: first, "whether mentioned," i.e., whether information related to the CDE exists in the abstract; second, "whether the value is correct," i.e., whether the extracted fact matches manual annotation. The 79.2% processing success rate corresponds to the first level, while the 94.9% exact match rate corresponds to the second level; the two have different meanings and should not be directly compared.
94.9% extraction agreement and error concentration domains
In 100 sampled abstracts, for 1,116 non-empty answer pairs, the exact match rate between LLM-extracted values and manually extracted values was 94.9% (abstract-level bootstrap 95% CI, 93.0–96.4%). This is a notably high figure, indicating that given defined CDEs, the agent's value-level extraction is approaching usable levels.
Error distribution is uneven. The study reports that errors concentrate in anatomical fields and temporal fields. These two field types precisely depend on contextual disambiguation and clinical narrative understanding, such as hierarchical attribution of anatomical sites and relative versus absolute conversion of temporal expressions. This points the way for future improvements: introducing ontology constraints or dedicated validation modules for anatomical and temporal dimensions may be more effective than general prompt engineering.
Exact match is a strict evaluation metric in information extraction: a prediction is counted as correct only when the model's output string (after normalization) is identical to the manual annotation; partial matches receive no credit. A 94.9% exact match rate means that among 1,116 non-empty answer pairs, the vast majority of extraction results exactly match manual values. Note that exact match is sensitive to field type: categorical fields (e.g., "yes/no") easily achieve high scores, while free-text or composite fields (e.g., anatomical site descriptions, temporal expressions) are harder to match exactly. Therefore, despite the high metric, its practical usability must be interpreted alongside the error distribution.
Implications for clinical data governance
Multiple sources of information in this study point to the same judgment: agents can significantly accelerate CDE schema drafting and large-scale extraction, but expert curation cannot be omitted. From a data governance perspective, the 79.2% processing success rate and 56.6% correctness approval rate together form a realistic baseline for a "human-AI collaboration pipeline"—the agent handles scale, and experts handle definition quality.
For teams planning to adopt a similar workflow, a reasonable implementation order is: first use agents to induce candidate CDEs from domain corpora, then apply dual filtering by frequency and expert review, and finally validate extraction agreement on a controlled corpus, with separately designed validation rules for high-risk fields (anatomy, time).
Agentic consolidation refers to having large language model agents cluster, merge, and deduplicate large numbers of candidate concepts, consolidating semantically similar candidates into a single CDE. In this study, 72,909 candidates were compressed to 1,570, a compression ratio of about 46:1. The core challenge of this step is determining whether "two candidates express the same clinical concept"—for example, "primary site" and "tumor location" may need merging, while "age at diagnosis" and "age at onset" must be carefully distinguished. Consolidation quality directly determines the burden of subsequent frequency filtering and expert review, and is one of the steps in the agent pipeline most dependent on prompt design and domain constraints.
One-line Summary
LLM agents automatically induce CDEs from sarcoma case abstracts and extract evidence, with high relevance but insufficient correctness.
Paragraph Summary
This study designed a two-stage large language model agent pipeline that proposed 72,909 candidate Common Data Elements (CDEs) from 24,588 PubMed sarcoma case report abstracts, compressed them to 1,570 via agentic consolidation, and retained 53 CDEs supported by at least 50 abstracts for large-scale evidence extraction. All retained CDEs were successfully processed in 79.2% of abstracts. Manual review showed that two reviewers' approval rates for CDEs were 83.0% for relevance, 67.9% for clarity, and only 56.6% for correctness. Among 1,116 non-empty answer pairs from 100 sampled abstracts, the exact match rate between LLM extraction and manual extraction reached 94.9% (95% CI 93.0–96.4%), with errors mainly concentrated in anatomical and temporal fields. The authors conclude that agents can accelerate CDE drafting and extraction, but expert curation cannot be omitted.
Related articles

FBI Warns About AI Hacking Tool Artex: Systems Breached in Three Hours
FBI warns about AI hacking tool Artex: describe targets in natural language for automated scanning and intrusion, with some attacks achieving full privileged access within three hours. It affected over 2,000 Microsoft 365 organizations and South Korean banks. Copies still circulate after the developer took it down in October; security fundamentals remain key.

FBI-Warned AI Hacking Tool ARTEX: Cost Disruption and Defensive Restructuring
The FBI-warned AI hacking tool ARTEX reduces penetration testing costs to $150-$1,500, causing financial data breaches and supply chain attacks. Learn how it works, real cases, and a five-step defense strategy.

FBI Warns: ARTEX AI Hacking Tool Breaches Systems in Three Hours
FBI warns the open-source AI pentesting framework ARTEX is being abused, with some intrusions gaining privileged access in under three hours, over 1,600 consoles exposed, and links to South Korean financial data breaches and the ConnectWise supply chain attack.