ICU Mortality Prediction Models: How Measurement Timing Affects Transportability

Introduction: The Transportability Challenge of Prediction Models

In the intensive care unit (ICU), there has been extensive research on using machine learning to predict patient mortality risk. However, one often-overlooked question is: why do models that perform well at the source hospital experience performance degradation after being transported to a new hospital? A study based on multi-center data from MIMIC-III, MIMIC-IV, and eICU systematically distinguished the predictive value of two types of features—physiological trajectories and observation patterns—and revealed the independent impact of measurement timing on model transportability.

Study Design and Feature Stratification

The study predicted in-hospital mortality at the 48-hour mark, using nested representations to progressively integrate three types of information:

• R1: Latest-value snapshots only
• R2: Adding longitudinal physiological summaries
• R3: Further incorporating measurement frequency and timing features

The model was fitted on 7,762 MetaVision patients, then externally evaluated on 21,682 MIMIC-IV patients and 43,878 eICU patients across 204 hospitals. Crucially, the study examined the impact of measurement timing alone while holding the number of retained observations constant.

Nested representations refer to a modeling strategy in which model input features are progressively layered according to information complexity and level of abstraction. In this study, R1 contains only latest-value snapshots, representing a "static cross-sectional" representation; R2 adds longitudinal physiological summaries (such as trend slopes, fluctuation ranges, and cumulative exposure) on top of R1 to characterize physiological states evolving over time; R3 further incorporates measurement frequency and timing features, reflecting the data collection process itself. The purpose of this layered design is to decouple two types of information—"patient physiological state" and "observation process"—so that their transportability can be evaluated separately in external validation. Nested representations are commonly used in electronic health record (EHR) modeling because EHR data contain both clinical signals and recording behavior, and confounding between the two can affect attribution analysis of model generalization capability.

Core Findings: Physiological Trajectories Are More Stable, Observation Patterns More Sensitive

Transportability of Physiological Trajectories

Physiological trajectory features (R2 relative to R1) brought consistent improvement in external eICU prediction, with a log-loss difference of -0.00922 (95% CI -0.01148 to -0.00704). This improvement remained stable across cross-institutional transport, indicating that longitudinal physiological changes carry relatively universal prognostic signals.

Fragility of Observation Patterns

Observation pattern features (R3 relative to R2) were also effective on the locally developed eICU model (-0.00257, 95% CI -0.00370 to -0.00146), but the effect was no longer clear after transport from MetaVision (-0.00012, 95% CI -0.00206 to +0.00208). In 30 bootstrap refits on the source data, only 10 favored R3. This indicates that measurement frequency and timing features are highly dependent on institution-specific monitoring practices and tend to fail when transported across hospitals.

Log-loss is a commonly used evaluation metric for binary classification prediction models, defined as the negative log-likelihood of the predicted probability under the true label, with lower values being better. Unlike AUC, log-loss penalizes both misclassification and overconfident probability estimates, and is sensitive to model calibration quality; therefore, it is often used to compare the degree of improvement different feature sets provide to the predicted probability distribution. This study reports paired log-loss differences (e.g., R2 relative to R1), where negative values indicate that the new feature set reduced loss and improved predictive performance. A 95% confidence interval not crossing zero is considered statistically significant. In cross-institutional external validation, the stability of log-loss differences reflects the actual calibration performance of the transported model better than a single-point AUC, which is also the reason the study chose this metric as the primary evaluation basis.

Independent Effect of Measurement Timing

Through experiments matching the number of observations, the study disentangled the confounding between "measuring more" and "measuring wisely":

• When the number of observations was matched, spreading measurements across the entire time window, compared with random retention, yielded a log-loss improvement of 0.03280 for R3, while R2 was only 0.00703.
• Removing measurements from the last 12 hours, compared with count-matched random removal, increased log-loss by 0.01947 for R3 and 0.00822 for R2.

This indicates that measurement timing itself carries independent predictive information, and the R3 model relies more heavily on it. Once the measurement rhythm in a new environment differs, this reliance translates into performance loss.

Bootstrap refitting is a model stability assessment method based on resampling. In this study, 30 bootstrap resamples were drawn from the source data, the model was refitted each time, and the incremental effect of R3 relative to R2 was evaluated on external data, counting how many times R3 was supported (i.e., the increment was beneficial). Only 10 out of 30 favored R3, meaning that the incremental advantage of observation pattern features is not robust under perturbations of the source data and is close to the random level. The value of bootstrap refitting lies in revealing whether "conclusions drawn from a single split" depend on a particular sample composition, and it is especially suitable for scenarios with limited sample size or weak feature effects. Compared with the point estimate from a single external validation, the bootstrap distribution more intuitively reflects the reproducibility of incremental features and provides a basis for feature selection.

Reversal When Availability Declines

The study further found that under conditions of reduced measurement availability, the incremental effect of local R3 shifts from beneficial to harmful. This suggests that observation pattern features are not "the more the better"; their value is tightly bound to data collection conditions.

Implications for Clinical Prediction Model Development

This study conveys a clear methodological signal: evaluation of clinical prediction models should not focus only on distribution differences in new patient populations, but should also systematically test the measurement conditions that supply model inputs. Specifically:

1. Prioritize developing physiological trajectory features that are insensitive to measurement rhythm to improve cross-institutional robustness.
2. Treat measurement frequency and timing features as "local enhancements" and report their increment separately in external validation.
3. Before model deployment, simulate the target institution's measurement protocol to assess performance fluctuations caused by changes in feature availability.

Conclusion

The transportability of ICU mortality prediction depends not only on differences in patient populations but also on the consistency of measurement conditions. Physiological trajectories provide more stable cross-hospital signals, while observation pattern features are a double-edged sword—effective at the home institution, yet potentially failing during transport due to changes in measurement timing. Model evaluation frameworks need to treat "measurement conditions" as a first-class citizen in their considerations.

One-line Summary

Physiological trajectory features are stable across hospitals in ICU mortality prediction, while observation pattern features readily fail upon transport.

Paragraph Summary

Based on multi-center data from MIMIC-III, MIMIC-IV, and eICU, this study predicted in-hospital ICU mortality at the 48-hour mark, stratifying features into latest-value snapshots (R1), longitudinal physiological summaries (R2), and measurement frequency and timing (R3), and systematically evaluated their transportability. Results showed that physiological trajectory features (R2 relative to R1) brought stable improvement in external eICU prediction (log-loss difference -0.00922), whereas observation pattern features (R3 relative to R2) were effective locally (-0.00257) but showed unclear increments after transport (-0.00012), with only 10 of 30 bootstrap iterations supporting R3. Experiments matching the number of observations further indicated that measurement timing itself carries independent predictive information, and spreading measurements outperforms random retention; when measurement availability declines, the incremental effect of R3 even shifts from beneficial to harmful. The study suggests that model evaluation should incorporate measurement conditions into consideration.