Textbooks or Patient Records? How Data Types Shape Medical LLM Capabilities

Medical large language models (Medical LLMs) are typically trained on a mix of two data types: didactic data represented by textbooks, and clinical data represented by patient records. A long-standing open question is — how do these two data types differentially shape model capabilities? A newly published arXiv study provides some compelling answers through rigorous controlled experiments.

Research Design: Token-Matched Controlled Experiments

The core of this study lies in token-matched experiments. By keeping the total number of training tokens constant, researchers systematically varied the ratio of didactic to clinical data, cleanly isolating the effect of "data type" as a variable without letting differences in data volume confound the results.

The experiments spanned two categories of downstream tasks: knowledge-intensive tasks — such as medical knowledge Q&A, testing the model's grasp of factual medical knowledge — and clinic-oriented tasks — such as reasoning over Electronic Health Records (EHR), testing the model's ability to perform in real clinical scenarios. This task stratification is key to understanding the findings that follow.

Electronic Health Records (EHR) are digitized records of a patient's complete medical information, encompassing diagnoses, lab results, medication history, surgical records, and clinical notes. Unlike the structured narrative of textbooks, EHR data is highly unstructured, filled with clinical abbreviations, non-standard expressions, and heavy contextual dependencies — which is precisely what distinguishes clinical data from didactic data. EHR-based reasoning tasks require models not only to possess medical knowledge, but also to understand the language patterns and decision logic of real clinical workflows, making them a widely used benchmark for assessing the practical clinical utility of medical AI.

Key Finding: Asymmetric Transfer Between Data Types

The study's most important finding is the presence of asymmetric transfer between data types.

Specifically, clinical data not only significantly improved performance on clinic-oriented tasks, but also remained highly competitive on knowledge-intensive tasks. Didactic data, by contrast, primarily boosted knowledge-intensive tasks while offering limited benefit for clinical tasks.

In other words, clinical data has a larger "generalization radius" — it can teach models how to reason in real-world scenarios without noticeably falling behind on pure knowledge assessments. Didactic data, on the other hand, behaves more like a specialist that excels at memorizing facts but struggles to transfer that knowledge to real clinical decision-making.

The Knowing-Doing Gap

Through error analysis, the study reveals an intriguing phenomenon the authors call the knowing-doing gap: improvements in knowledge recall do not reliably generalize into gains in clinical reasoning ability.

This finding raises a cautionary flag for how medical AI is evaluated. The field commonly uses medical exam benchmarks (such as USMLE-style tests) to measure the performance of medical LLMs, but a high score does not mean a model can make correct judgments in real clinical cases. There is a gap between knowing medical facts and practicing medicine — one that cannot be bridged simply by piling on more textbook data. This also explains why many models that shine on standardized tests still struggle when deployed in clinical decision-support settings.

USMLE (United States Medical Licensing Examination) is one of the most commonly cited evaluation benchmarks for medical LLMs. The exam spans three steps covering basic medical science, clinical knowledge, and clinical skills, and is primarily multiple-choice. Its standardized format and unambiguous answers make it well-suited for quickly assessing a model's medical knowledge, and mainstream models like GPT-4 have already reached or exceeded the human average on this benchmark. However, USMLE remains fundamentally a knowledge-based assessment — its scenarios are highly simplified compared to the ambiguous information, multi-system interactions, and decision uncertainty found in real clinical records. This is exactly how the "knowing-doing gap" manifests at the evaluation level: high USMLE scores largely reflect knowledge recall ability, not actual clinical reasoning.

A Small Amount of Clinical Data Can Capture Most of the Gains

Another practically valuable observation is that a modest amount of clinical data can deliver the majority of gains on EHR-related tasks. This means that even when clinical data is expensive to obtain and subject to strict privacy constraints, teams don't need to pursue massive clinical corpora — at the right mixing ratio, a small amount of high-quality clinical data yields extremely high marginal returns.

The study also emphasizes that the optimal mixing ratio is not fixed, but varies with how much each downstream task demands knowledge versus clinical reasoning. Reasoning-intensive applications benefit from a higher proportion of clinical data, while knowledge retrieval applications can rely more heavily on didactic data.

The scarcity and high cost of clinical data is a long-standing core bottleneck in medical AI. On one hand, patient records are strictly protected by privacy regulations such as HIPAA (Health Insurance Portability and Accountability Act), and de-identification is both costly and carries re-identification risks. On the other hand, real clinical data is often siloed across healthcare institutions, making aggregation difficult. This gives the principle of diminishing marginal returns particular practical significance for clinical data — if a small amount of carefully curated clinical text can capture most of the capability gain, teams can concentrate their limited data acquisition resources on quality rather than quantity, while using didactic data to reduce dependence on large volumes of clinical text.

Implications for Medical LLM Data Curation

Synthesizing these results, the study's central recommendation is that data curation for medical LLMs should be application-driven, rather than chasing data scale or some universal "standard recipe."

• For use cases involving real clinical practice and intensive reasoning, prioritize a higher proportion of clinical data;
• For knowledge retrieval and health information Q&A use cases, didactic data remains an efficient choice;
• Evaluation should go beyond knowledge-based benchmarks and incorporate clinical reasoning assessments to truly reflect a model's deployment readiness.

For teams building medical vertical LLMs, this work provides a practical framework: first clarify how much the target application weighs "knowledge" versus "reasoning," then design the data mix accordingly — rather than stuffing every available corpus into the training set. The type and ratio of data may matter more than sheer volume in determining a model's clinical value.