Building a Leakage-Proof Diabetes Readmission Prediction Model: A Practical Guide on 100K Hospital Visits

Title

Building a Leakage-Proof Diabetes Readmission Prediction Model: A Practical Guide on 100K Hospital Visits

Meta Description

A complete workflow for building a readmission prediction model on the UCI diabetes dataset: patient-grouped splits, avoiding None-as-missing errors, column-by-column target leakage checks, Pipeline-wrapped preprocessing, and reporting PR-AUC and calibration instead of accuracy alone.

Article Content

Hospital readmissions are a heavy burden on the healthcare system, especially for diabetes patients—a return visit shortly after discharge often signals gaps in glycemic control, medication handoff, or follow-up care. If high-risk patients can be identified in advance, limited follow-up resources can be directed where they matter most. An author with a clinical background walked through the full modeling workflow using the public UCI Diabetes 130-US Hospitals dataset (101,766 inpatient records from 130 US hospitals), but the core of the process was not hyperparameter tuning—it was how to define the problem, split the data, and evaluate honestly.

Understand your rows first: each visit is not each patient

The dataset has 101,766 rows but only 71,518 unique patients, and repeat-admission patients contribute 46.2% of the rows. This fact directly determines the evaluation design: if records from the same patient appear in both training and test sets, the model partly "remembers" that person rather than learning generalizable patterns. The author found that a naive random split would place 6,791 patients in both sets at once, with 41.6% of test visits coming from patients the model had already seen. Patient-grouped splitting is therefore not optional—it is mandatory.

The data-loading trap: None and missing values are not the same thing

The dataset marks missing values with ?, but in the HbA1c and glucose columns the text None means "test not performed," which is a real clinical category. pandas by default converts None to a missing value, so the author's data initially lost all 96,420 records of "no glucose test performed." The correct loading approach is na_values=['?'] combined with keep_default_na=False, preserving the clinical meaning. In addition, missingness hides in integer codes too: admission types 5, 6, and 8 represent not available, NULL, and not mapped respectively, covering 10.2% of visits, and these need to be converted to missing using the dataset's bundled ID mapping file.

pandas missing-value parsing mechanism: By default, read_csv automatically recognizes a set of predefined strings (such as 'NA', 'NULL', 'None', etc.) as missing values. Setting keep_default_na=False disables this behavior, and na_values=['?'] then precisely specifies which strings should be treated as missing. In medical data, distinguishing "test not performed" from "data missing" is critical: the former is a real clinical state that carries information; the latter is a data-entry problem. Keeping None as a categorical level lets the model exploit the pattern of "not tested," while true nulls must be handled via imputation. This is a common but easily overlooked semantic trap at the data-loading stage.

Define the outcome and prediction timepoint: more important than model choice

The readmitted column has three values: <30 (within 30 days), >30 (later), and NO. The author split this into two independent binary classification tasks, never mixing them: any readmission (primary task, 47.1% positive rate) and 30-day readmission (secondary task, 11.4% positive rate). The prediction timepoint is fixed at the end of the hospital stay, before discharge. The author also removed 2,423 visits that ended in death or hospice care, and the data confirmed that all 1,652 death outcomes were labeled NO; keeping them would let the model learn meaningless rules.

Check for leakage column by column: discharge disposition may only define the cohort

Target leakage is the silent killer of medical models. The author wrote down the rationale for including or excluding each variable. Discharge disposition is used only when defining the cohort, not as a predictor, because it is determined at the very end of the stay. Payer code (billing data, 40% missing) and weight (97% missing) were excluded. Primary diagnosis codes were grouped into nine coarse clinical categories, under the assumption that the working diagnosis is known before discharge. Secondary diagnosis codes are usually finalized by coders after discharge, so the first version of the model did not include them. The code also adds an assertion: if any forbidden column enters the feature matrix, it fails immediately.

Target Leakage refers to information mixed into the training features that is only available after the prediction timepoint or is directly related to the outcome, causing the model to look artificially strong on validation data but fail after deployment. Common manifestations include using codes finalized after discharge, including derived variables of the outcome itself, or directly using billing settlement data. The way to check is to first define the prediction timepoint and then ask, column by column, "is this information known at that timepoint?" In this case, removing death or hospice visits, excluding secondary diagnosis codes, and restricting discharge disposition to cohort definition only are all concrete measures against leakage. Adding an assertion in code that forbids certain columns from entering the feature matrix is an enforceable safeguard against leakage.

Use Pipelines and patient-grouped splits to guarantee honest evaluation

The author used StratifiedGroupKFold to make an 80/20 train/test split grouped by patient while keeping the readmission rate balanced (0.471 on both sides), and asserted zero patient overlap. All preprocessing that requires learning was placed inside scikit-learn's Pipeline and ColumnTransformer: median imputation, standardization, one-hot encoding with rare-category merging (min_frequency=100, handle_unknown='infrequent_if_exist'). No transform was ever fitted on test data, including during cross-validation.

outer_split = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_idx, test_idx = next(outer_split.split(X, y, groups))
assert not set(groups.iloc[train_idx]) & set(groups.iloc[test_idx])   # zero shared patients

StratifiedGroupKFold is a splitter in scikit-learn for grouped cross-validation. It preserves the distribution of the target variable across folds (stratified) while ensuring that all samples from the same group (such as the same patient) appear in only one fold (group). The grouping is specified by the groups parameter, so the patient identifier must be passed when calling split. In data with repeat visits, this forces the patient sets in training and test to be mutually exclusive, preventing the model from inflating evaluation scores by "remembering" individuals rather than learning patterns. It is suitable for scenarios where reliable generalization performance must be reported, but it may increase variance between folds, so it is often used with a fixed random seed.

Baselines and models: accuracy can deceive

The majority-class baseline reaches about 52.9% accuracy yet identifies not a single readmitted patient—a reminder that accuracy is not a clinical metric. Logistic regression achieves a test-set ROC-AUC of 0.679 (patient-level bootstrap 95% confidence interval 0.670–0.687) and a PR-AUC of 0.651, with a positive rate of 0.471. Training-set cross-validation is very stable (0.672 ± 0.003). A histogram gradient boosting model reaches ROC-AUC 0.691 (95% CI 0.683–0.700), with about 10 seconds of training cost.

Is the boosted model really better than logistic regression? The author argues there is no substantive difference: the confidence intervals overlap, AUC improves by only 0.013, and specificity at the same sensitivity improves by only 2.3 percentage points. The signal comes mainly from one nearly monotonic feature—the number of inpatient visits in the past year. Shuffling that feature drops ROC-AUC by 0.075, roughly four times the impact of other features. Given transparency considerations, the author ultimately kept logistic regression.

Thresholds and calibration: catching 70% of readmissions means following up half of discharged patients

The decision threshold was chosen on out-of-fold training predictions, not on the test set, with the rule: take the highest threshold subject to sensitivity of at least 70%, yielding 0.428. At that threshold, the test set has sensitivity 0.697, specificity 0.547, precision 0.579, and would flag 56.8% of patients. The rule transfers almost losslessly from the training set (70.0%) to new patients (69.7%). The honest interpretation: to catch 70% of readmissions, the team needs to follow up more than half of discharged patients.

The calibration curve hugs the diagonal. The mean predicted risk is 0.469 versus an observed 0.471, the calibration slope is 1.05, and every decile is within 5 percentage points of the observed rate. For discharge teams, calibration matters: a "60% risk" label should mean that roughly six out of ten people will actually come back. In this population, the model delivers.

Calibration measures how well predicted probabilities match observed frequencies, which is different from discrimination (such as AUC). A model with high discrimination but poor calibration may rank risk correctly yet give misleading absolute probabilities. Common checks include calibration curves, calibration slope (ideally 1), and comparison of mean predicted risk with the observed rate. In clinical decision-making, calibration directly affects threshold selection: if the model predicts "60% risk" but only 40% of people are actually readmitted, resource allocation based on that probability will be misaligned. Methods such as Platt scaling and isotonic regression can be used for post-hoc calibration, but if calibration is already good, deployment communication can be simplified.

A few tips for beginners

• Spend the first few hours on the data and the problem, not the algorithm.
• Before choosing features, write down the prediction timepoint.
• When patients have repeat records, split by patient.
• Put all learned transforms inside a Pipeline.
• Report PR-AUC, calibration, and threshold tables—not just accuracy.
• Be explicit about what the model cannot do: this model uses 1999–2008 US data, and without local validation it cannot be used for patient care in other regions.

The complete code, figures, and report are on GitHub: https://github.com/Mwaivictor/diabetes-readmission-prediction. The dataset comes from Clore, Cios, DeShazo, and Strack (2014), UCI Machine Learning Repository, CC BY 4.0 (DOI 10.24432/C5230J). Original study: Strack et al. (2014), BioMed Research International.

One-line Summary

A clinician author uses the diabetes readmission dataset to demonstrate a full modeling workflow covering problem definition, patient-grouped splitting, and honest evaluation.

Paragraph Summary

Based on the UCI diabetes readmission dataset (101,766 inpatient visits, 71,518 patients), the article walks through the full modeling workflow, but the core is not tuning—it is problem definition, data splitting, and honest evaluation. The author emphasizes patient-grouped splitting (46.2% of rows come from repeat-admission patients, and a naive random split would let 41.6% of test visits come from patients the model has already seen), preserving the clinical meaning of "test not performed," defining features at the pre-discharge timepoint, and checking leakage column by column. Model comparison shows no substantive difference between logistic regression (ROC-AUC 0.679) and gradient boosting (0.691), with the signal mainly from the number of inpatient visits in the past year. At a 70% sensitivity threshold, 56.8% of discharged patients must be flagged, and calibration is good—but the model is based only on 1999–2008 US data, so external use requires local validation.