Data Leakage

Data leakage refers to test set information being indirectly seen by the model during training, causing evaluation metrics to be inflated; it is one of the most common and most concealed methodological flaws in machine learning research, and is especially dangerous in the medical prediction domain.

Data leakage falls into two categories—target leakage and training set leakage—and causes models to perform well during testing but fail after deployment.

The BUG:/DOC:/ENH: prefixes commonly found in Issue titles cause data leakage, because the model would directly "copy the answer" from the title prefix rather than learn from the report content, so these prefixes must be removed before training.

When there is speaker overlap between the training set and test set, an accent classification model may learn to recognize specific speakers rather than accent features, leading to inflated evaluation results.

The retention column in the dataset is only populated after a customer initiates a cancellation, making it a data leakage feature that should not be used for training.

The solution to the data leakage is to have the agent perform training on an isolated branch restored to the exact timestamp prior to the prediction window.

The root cause of the issue in this case was data leakage in the data split, where random assignment placed nearly duplicate form templates on both the training set and regression test set sides simultaneously

If data augmentation is performed before splitting the dataset, and multiple images derived from the same original image end up in both the training set and the validation set, data leakage will occur.

The way to avoid data leakage is to split data by individual — all augmented images derived from the same original image must go either entirely into the training set or entirely into the validation set.

Other common sources of data leakage include computing mean/standard deviation for normalization using the full dataset, using future time-point information in feature engineering, and failing to apply cross-validation isolation during target encoding.

Data leakage almost never causes an error; the code runs normally and the metrics look great, making it one of the hardest-to-automatically-detect pitfalls in AI-generated code.

In AutoML tasks, DeepSeek Flash can autonomously formulate plans, perform feature selection, run multiple sets of hyperparameter experiments, and proactively identify and eliminate features with data leakage.