Evaluation Pitfalls in Machine Learning Rainfall Prediction: Climate Baselines Are Hard to Beat

Machine learning often overestimates performance in rainfall prediction due to improper evaluation methods. A study based on 35 weather stations in Bangladesh with 618,650 station-day records (1948–2014) systematically compared three evaluation approaches—random splitting, temporal splitting, and spatiotemporal splitting—and compared calendar-only features with features including lagged rainfall, six learners with climatology, persistence, and SARIMA models. The study found that a simple station-month climatology baseline achieves 88.4% accuracy (AUC 0.950) in classifying high-rainfall months, and under temporal evaluation, no learner can significantly outperform it.

Optimistic Bias of Random Splitting: Models See the Future

Randomly shuffling station records for training and testing allows the model to see observations from neighboring stations on the same day when predicting a given day, while also obscuring the fact that a seasonal lookup table already explains most of the variance. In the study, a random forest using only calendar features achieved a next-day wet-day AUC of 0.889 under random splitting, higher than any model using real predictors under temporal evaluation. However, when transferred to future years, its AUC dropped to 0.835, below the climatology baseline's 0.859. This shows that random splitting severely overestimates a model's generalization ability, especially when the model relies on calendar features—it is effectively memorizing seasonal patterns rather than learning predictive signals.

Data leakage refers to the inclusion of information in training data that should not be available at prediction time, leading to overly optimistic evaluation results. In spatiotemporal data, randomly splitting training and test sets causes observations from adjacent dates at the same station or from neighboring stations on the same date to appear in both training and test sets, allowing the model to "peek" at spatiotemporal neighborhood information of test samples. This leakage is particularly insidious in time series and spatial data because samples are not independent and identically distributed. The correct approach is to use temporal splitting (splitting by chronological order) or spatiotemporal splitting (considering both temporal and spatial isolation) to ensure that test set information is completely invisible during training.

Strength of the Climatology Baseline: Simple but Hard to Beat

Station-month climatology, using only historical averages, can classify high-rainfall months with 88.4% accuracy and an AUC of 0.950. Under temporal evaluation (training on data up to 1999, testing on 2005–2014), no learner improved over climatology by more than 0.8 percentage points. For monthly total rainfall, no learner significantly outperformed climatology either. This result challenges the assumption that machine learning is necessarily superior to simple baselines in rainfall prediction, suggesting that researchers must treat climatology as a minimum baseline and report skill scores relative to it.

The climatology baseline uses long-term historical averages (such as station-month mean rainfall) as predictions. It does not rely on any real-time predictors and reflects only seasonal cycles and local climate characteristics. In rainfall prediction, the climatology baseline is strong because rainfall seasonality and spatial differences dominate the variance, and a simple lookup table can explain most of the predictable signal. Therefore, any machine learning model that cannot significantly outperform the climatology baseline has questionable practical value. Reporting skill scores relative to climatology (such as the Brier skill score) has become standard practice in meteorological forecast evaluation.

Lagged Rainfall Features Provide Real but Limited Skill

Models including lagged rainfall features show modest but real skill in next-day prediction: a Brier skill score of 0.074 (95% CI 0.064–0.088) relative to climatology. When station and year inputs were removed, the skill score rose to 0.091; when same-day observations from other stations were added, it rose to 0.131. However, the GRU sequence model performed comparably to gradient boosting but did not surpass it, and skill disappeared beyond two days of lead time. This indicates that the predictability window for rainfall is short, and complex models may not bring additional gains.

The Brier skill score (BSS) measures the improvement of a probabilistic forecast relative to a reference forecast (here, climatology), calculated as 1 - BS_model/BS_ref, where BS is the Brier score (mean squared error between predicted probabilities and observed binary outcomes). A BSS greater than 0 indicates the model is better than the reference, equal to 0 indicates no improvement, and less than 0 indicates worse than the reference. BSS is sensitive to rare events and should be interpreted with confidence intervals. The BSS reported in the study ranges from 0.07 to 0.13, which is statistically significant but limited in magnitude, indicating a short window of true predictability.

Leakage-Aware Protocol: Promoting More Rigorous Evaluation

The research team released a leakage-aware evaluation protocol and code, requiring future station-based studies to report skill relative to climatology on held-out years. This initiative helps curb inflated performance caused by data leakage and promotes reproducibility and practical value in rainfall prediction research. For practitioners, using temporal or spatiotemporal splitting to evaluate models and always comparing against a climatology baseline is a key step to ensuring models are genuinely useful.

Spatiotemporal splitting is an evaluation strategy requiring that training and test sets do not overlap in either time or space. For example, training on data from earlier years and some stations, and testing on data from later years and other stations. This simultaneously avoids temporal leakage (future information flowing into training) and spatial leakage (neighboring station information flowing into training), more realistically simulating operational forecasting scenarios. Spatiotemporal splitting is generally stricter than temporal splitting alone, especially in regions with dense stations and strong spatial correlation. Researchers should choose an appropriate splitting method based on the actual application scenario and clearly report the splitting strategy.