DischargeBench: Teaching LLMs to Explain Discharge Instructions to Patients

When Discharge Education Meets Large Language Models

Discharge education is fundamentally an interactive teaching task. Clinicians must translate complex discharge plans into content that each patient can understand and remember, tailored to their literacy level, memory capacity, and personality. This is fundamentally different from typical LLM evaluation scenarios — most existing benchmarks focus on static Q&A or document generation tasks, with almost no measurement of "how much does the patient actually understand during open-ended dialogue."

A recent study published on arXiv (arXiv:2609.20827) addresses precisely this gap with a new evaluation framework called DischargeBench. The researchers' core argument is straightforward: LLM evaluation for discharge education should center on "patient comprehension" rather than merely measuring text quality or answer accuracy.

How DischargeBench Works

DischargeBench uses a "persona-grounded" simulation mechanism. The evaluation involves three roles:

• Candidate LLM Educator: the model being evaluated, responsible for explaining the discharge plan to the patient;
• Virtual Patient: played by a model, with a specific persona configuration;
• Education Monitor Agent: responsible for maintaining the realism of the virtual patient without interfering with the educator itself, thereby preserving the integrity of the evaluation signal.

The clever design here is that the monitor agent only manages "how realistic the patient seems" — it does not modify the educator's behavior. This ensures that the model's true performance across multi-turn conversations is not contaminated by external intervention, making the evaluation results a more accurate reflection of its actual capabilities.

Dataset: MIMIC-IV-Ext-DischargeBench

The research team built the MIMIC-IV-Ext-DischargeBench dataset based on the well-known MIMIC-IV medical database, containing 477 cases spanning 24 ICD disease chapters. Each case is configured with multi-dimensional persona axes, including:

• Personality
• Education level
• Health literacy
• Past-medical-history recall ability

These dimensions allow researchers to perform stratified analysis — observing how models perform with different types of patients rather than simply reporting an overall average score.

MIMIC-IV (Medical Information Mart for Intensive Care IV) is a large de-identified clinical database jointly maintained by MIT and Beth Israel Deaconess Medical Center, containing over 400,000 ICU and inpatient visit records encompassing diagnoses, medications, lab results, clinical notes, and more. It is one of the most widely used public datasets in medical AI research, requiring CITI ethics training and a signed data use agreement for access. ICD (International Classification of Diseases) chapters are a disease classification system established by the World Health Organization; the tenth edition (ICD-10) organizes all diseases into 22 chapters by system, such as circulatory system diseases, neoplasms, and mental disorders. Using 24 ICD chapters as a coverage standard means the dataset aims for broad representativeness across the disease spectrum rather than concentrating on a single department or condition — which is critical for evaluating a model's generalization ability.

Four Dimensions for Measuring "Did It Get Through?"

Each simulated conversation is scored along four axes by an LLM-as-a-Judge aligned with physician annotations:

1. Conversation Quality: whether the interaction is smooth and appropriate;
2. Topic Checklist: whether the key points that should be covered are fully addressed;
3. Comprehension: whether the patient genuinely understood the content;
4. Factual Consistency: whether the explanation faithfully reflects the original discharge record.

The value of this multi-dimensional scoring system lies in breaking down "quality of teaching" into measurable, specific capabilities. A model might produce beautifully written text but perform poorly on the "patient comprehension" dimension — a problem that traditional evaluations rarely expose.

LLM-as-a-Judge is an approach that has emerged in NLP evaluation in recent years: using a powerful language model (such as GPT-4) as a substitute for human annotators to score or compare the outputs of other models. Its advantages include low cost, scalability, and relatively high correlation with human judgment on many tasks. Its limitations include positional bias in the judge model, self-preference (a tendency to rate stylistically similar outputs higher), and insufficient capability for fine-grained fact verification. In medical contexts, these limitations warrant particular caution — a model judge may fail to accurately identify subtle but clinically critical errors. This study attempts to mitigate the issue by calibrating the judge model against physician annotations, but the researchers acknowledge that its reliability still requires further validation — a threshold that must be crossed before this approach moves from academic evaluation to clinical deployment.

Key Finding: Aggregate Scores Mask Clinical Risk

The most noteworthy conclusion of the research is this: regardless of whether models are closed-source or open-source, aggregate scores mask clinically relevant differences across ICD chapters and across patient personas.

In other words, a model with a seemingly good overall score may have serious weaknesses for specific disease types or specific patient profiles. The study specifically highlights that difficult personas expose three types of model failures:

• Coverage failures — key topics that should have been explained are missed;
• Comprehension gaps — the patient didn't understand the content;
• Reduced source-answer agreement — the explanation diverges from the original discharge record.

This finding is particularly important in medical contexts. Patients with low health literacy, poor memory, or resistant personalities represent precisely the moments when high-quality communication is most needed — and yet models are most prone to failure in exactly these situations. Making deployment decisions based solely on average scores could introduce hidden clinical risks.

Health literacy refers to an individual's ability to obtain, understand, and use health information to make correct decisions. Data from the National Assessment of Adult Literacy shows that approximately 36% of adults have basic or below-basic health literacy, a proportion that is even higher among older adults, patients with chronic conditions, and low-income populations. Low health literacy is directly associated with adverse clinical outcomes including increased medication error rates, higher readmission rates, and reduced self-management ability. Discharge education is a critical intervention for bridging this gap — research shows that effective discharge education can reduce 30-day readmission rates by approximately 20%. This is precisely why DischargeBench incorporates health literacy as a persona dimension and places particular emphasis on model performance among low-literacy populations: the patients who are hardest to communicate with are often also the ones who most need that communication to be high quality.

Implications for Medical AI Deployment

The significance of DischargeBench extends beyond introducing a new benchmark. It proposes a methodological shift: when evaluating medical dialogue AI, "patient comprehension" should be the core metric, not text fluency or answer correctness as endpoints.

For teams exploring deployment in medical settings, this offers several practical takeaways:

• Static Q&A evaluations cannot represent real multi-turn communication capabilities;
• Stratified evaluation (by disease type, by patient profile) provides more diagnostic value than a single average score;
• Having models simulate virtual patients with different personas is a low-cost but high-information stress-testing approach.

Of course, as an initial study, DischargeBench also relies on the reliability of LLM-as-a-Judge, and its alignment with actual physician judgment still requires ongoing validation. But it at least provides an actionable starting point for answering the question: "How do we evaluate whether an AI can truly explain discharge instructions clearly?"