LLM-Extracted Features from Clinical Notes Improve Extubation Failure Prediction

The Clinical Challenge of Predicting Extubation Failure

Invasive mechanical ventilation is a life-saving intervention in critical care, but deciding when to safely wean patients off the ventilator remains one of the most difficult clinical judgment calls. Premature weaning can lead to extubation failure (EF), requiring reintubation — which not only raises the risk of infection and airway injury, but significantly prolongs hospitalization and increases mortality.

A research team from the University of Washington Medicine has proposed a new approach: using large language models (LLMs) to extract features from free-text respiratory therapy notes to improve extubation failure prediction. The core idea is to bring clinical narrative information — long overlooked by structured-data models — back into the modeling pipeline.

Extubation failure is clinically defined as the need for reintubation within 48 to 72 hours after extubation, with an incidence of approximately 10%–20%. Reintubation is far from a simple procedural repeat — studies show that ICU mortality in reintubated patients can be 6 to 8 times higher than in those successfully extubated, and each intubation attempt carries additional risks of airway trauma, aspiration pneumonia, and ventilator-associated pneumonia. Standard weaning assessment tools such as the Spontaneous Breathing Trial (SBT) and the Rapid Shallow Breathing Index (RSBI) capture a patient's current physiological state, but evaluating secretion management capacity, neuromuscular recovery, and subjective tolerance still depends heavily on bedside clinical judgment — judgment that is typically documented in free-text nursing or respiratory therapy notes and remains inaccessible to traditional predictive models.

Teaching LLMs to Read Respiratory Therapy Notes

Conventional EF prediction models rely primarily on structured patient data — vital signs, lab values, ventilator parameters, and the like. These inputs are clean and easy to compute, but respiratory therapists capture rich observational detail in their clinical notes — spontaneous breathing performance, secretion characteristics, mental status, and responses to weaning trials — that carries diagnostic signal that structured fields simply cannot encode.

The researchers built an "LLM + logistic regression" pipeline: an LLM classifies free-text respiratory therapy notes to identify features associated with extubation failure, and those features are then combined with structured data and fed into a logistic regression model. This hybrid approach preserves the LLM's strength in understanding unstructured text while leveraging logistic regression to maintain model interpretability — a critical requirement in clinical settings where clinicians need to understand why a model makes a particular prediction.

Why Logistic Regression Instead of an End-to-End Deep Model

In medical decision-making, interpretability often matters more than raw predictive accuracy. The coefficients of a logistic regression model intuitively reflect the direction and magnitude of each feature's contribution, allowing clinical teams to verify whether the model's logic aligns with medical reasoning. In this architecture, the LLM serves as a "feature engineering" layer — converting narrative text into quantifiable clinical features — rather than directly producing opaque black-box predictions.

Respiratory therapists (RTs) are ICU specialists responsible for managing mechanical ventilation and evaluating weaning readiness. Their clinical notes, typically written as unstructured free text, document cough strength, sputum viscosity and volume, SBT tolerance, and patient cooperation. The key distinction between these narrative records and numerical vital signs is that they integrate the therapist's subjective judgment with multi-dimensional observations — an important complement to structured data. Yet without a standardized coding scheme, these notes have long existed as data silos within electronic health record systems, inaccessible to machine learning pipelines. LLMs, with their capabilities in semantic understanding and zero-shot classification, now offer a viable path to systematically mining this type of documentation.

Two Core Findings

The first finding is that LLM-derived features extracted from respiratory therapy notes carry meaningful clinical signal. When combined with structured patient data, they improve extubation failure prediction performance — validating that free-text records contain useful information that traditional modeling approaches have left on the table.

The second — and perhaps more emphasized — finding is that performance discrepancies across prior EF prediction studies are largely attributable to inconsistencies in target populations. Different studies have applied heterogeneous inclusion criteria and varying definitions of extubation failure, leading to systematic differences in reported model performance that undermine cross-study comparability and generalizability.

The inconsistent definition of extubation failure is a long-standing methodological hazard in this field. Some studies define failure as reintubation within 24 hours of extubation; others use 48- or 72-hour windows. Inclusion criteria vary as well — some studies exclude unplanned or palliative extubations, others include them. This heterogeneity directly alters the base rate of positive events, which in turn affects sensitivity, specificity, and AUC. The lower the base rate, the more a model can appear "excellent" on surface metrics while offering little actual clinical utility. Because cohort construction methods differ across studies, even two papers reporting similar AUROCs may not be solving the same clinical problem — and a model trained at one institution cannot simply be deployed at another.

Implications for Clinical AI Research

The value of this work lies not only in proposing a better prediction method, but in surfacing a methodological issue that is easy to overlook. When different teams define "extubation failure" differently and apply different patient selection criteria, reported performance metrics — whether similar or divergent — cannot be directly compared. This is a call to the broader clinical prediction research community to establish more consistent standards for data definitions and cohort construction; without such standards, even high accuracy figures may represent only locally valid conclusions for a specific population.

For AI deployment in healthcare, this study also demonstrates a pragmatic technical architecture: rather than pursuing an end-to-end system designed to replace clinicians, let LLMs serve as feature extraction tools that transform large volumes of unstructured clinical text into usable signals, then hand off to transparent, auditable traditional models for final decision-making. This "LLM as assistant, interpretable model as decision-maker" architecture may be better suited to current medical regulatory requirements and clinical trust than pure large-model end-to-end approaches.

Limitations and Future Directions

As a single-institution study based on a University of Washington Medicine cohort, the external validity of these findings still needs to be verified across more centers and more diverse patient populations — which is precisely the concern the paper itself raises about generalizability. Future work that standardizes EF definitions across multi-center data and replicates this methodology would more compellingly demonstrate the universal value of LLM-derived features.

Overall, this is a rigorous exploration that meaningfully connects LLM capabilities with real clinical needs — offering both an actionable technical approach and raising evaluation standard questions that the entire field would benefit from reflecting on.