ChurnWatch in Practice: Building MLOps Data Drift Monitoring on AWS

Training a model is not the finish line. As real-world data evolves over time, the data fed into a model can gradually drift away from its training distribution—this is data drift. A model that performs brilliantly on the training set can silently degrade in production due to data drift.

ChurnWatch is a small MLOps project built around exactly this problem: it leverages the AWS ecosystem to create a complete pipeline from model training to drift detection to metric monitoring, demonstrating the core ideas of data drift monitoring at minimal cost.

Why data drift deserves attention

Imagine a company that has trained a churn prediction model on historical customer data to identify which customers might cancel their subscriptions. The model itself is fine, but over time, customers' spending amounts might shift up or down overall, the average number of support calls might change, and the overall customer profile shifts along with it.

This means: even if not a single line of the model file changes, the data flowing into it no longer looks like the training data. Model reliability suffers, and teams often have no idea. ChurnWatch's goal is to make this kind of change visible, quantifiable, and monitorable.

The overall pipeline is refreshingly clear: train the model → compare new data against training data → detect drift → monitor results.

Data drift can typically be further broken down into several categories: covariate shift, where the distribution P(X) of input features X changes while the conditional distribution P(Y|X) stays the same; label shift (prior shift), where the class prior P(Y) changes; and concept drift, where P(Y|X) itself changes—meaning the same input corresponds to different true labels at different times. ChurnWatch uses PSI to detect primarily the first two, since PSI only looks at feature distributions and doesn't involve labels. Distinguishing these types helps in choosing the right mitigation strategy: covariate shift can be mitigated through resampling or reweighting, concept drift often requires retraining the model, and label shift calls for adjusting output thresholds or class priors.

ChurnWatch's technical architecture

The project's design is deliberately restrained, using only the most fitting AWS service at each stage:

• Amazon S3: stores the training dataset as the foundational data management layer
• Amazon SageMaker: trains the churn prediction model using historical customer data
• Python drift detection script: uses the Population Stability Index (PSI) to compare baseline training data with simulated production data
• Amazon CloudWatch: ingests drift metrics and provides continuous monitoring

The dataset is intentionally split into two parts: a baseline/training dataset and a simulated production dataset. The production dataset is deliberately constructed with a different feature distribution, giving the monitoring system a real drift scenario to detect. This isn't an idle demo—it's a full validation with an adversarial edge.

From training to detection: how PSI quantifies drift

The training stage is relatively standard: training data lives in S3, and SageMaker reads historical customer data to train the model. The truly interesting part is drift detection.

ChurnWatch uses the Population Stability Index (PSI) as its core metric. PSI compares the distribution of a feature in the baseline dataset versus a new dataset, producing an interpretable number. A simplified interpretation guideline is:

• PSI < 0.10: virtually no significant change
• PSI reaching 0.20: significant drift

In the experiment, production data is deliberately generated with a different distribution. The detection script calculates PSI for each selected feature and identifies the largest drift value. The output looks something like: max PSI is some specific value, threshold set at 0.2, verdict is "drift detected." The actual value depends on the generated production dataset.

PSI's value lies in this: it doesn't depend on the model itself, only on data distributions, so it can raise an alert before model inference even happens.

PSI is calculated by binning continuous features (typically 10 intervals), counting the proportion of samples in each interval for both the baseline and new datasets, then summing (actual proportion − expected proportion) × ln(actual proportion / expected proportion) across all bins. Mathematically it resembles a symmetrized KL divergence and is fairly sensitive to relative distribution changes. A few practical caveats: bin boundaries must follow the baseline data's boundaries, or results aren't comparable; each bin shouldn't have too few samples, or estimation noise gets amplified into PSI; PSI needs a smoothing term for zero-proportion bins to avoid logarithmic divergence. Additionally, PSI only handles numeric or binned categorical features, so high-dimensional sparse or text features need dimensionality reduction or embedding aggregation first.

Turning drift into monitorable metrics with CloudWatch

Once drift computation is complete, results are published to Amazon CloudWatch. This step transforms a one-off script output into a continuously observable metric.

Compared to manually running a script to check drift each time, CloudWatch changes things at the operational level: the drift value exists as a time-series metric that can be viewed, compared, and further wired into alarms. ML workflows now have observability capabilities similar to traditional service monitoring.

This is one of ChurnWatch's key takeaways—ML systems need not just training scripts, but a monitoring layer aligned with production operations.

CloudWatch custom metrics are written via the PutMetricData API, and each data point includes a namespace, metric name, dimensions, timestamp, and value. When publishing PSI as a metric, it's advisable to use "feature name," "dataset version," and "model version" as dimensions, so you can filter and aggregate by dimension in the console and more easily configure CloudWatch Alarms later. Alarms can trigger downstream actions like email, Lambda, or PagerDuty via SNS, closing the loop of "notify when drift exceeds threshold." Compared to writing if-else print statements in a script, the advantage of metrics is that they naturally preserve time-series history, allowing trend charts, period-over-period comparisons, and side-by-side comparison with other service metrics on the same dashboard.

The mindset shift from "building a model" to "operating a model"

The most valuable output of this project is actually a cognitive distinction the author arrived at through practice: building an ML model and operating an ML system are two different things.

Training a model is just one step in the lifecycle. Once a model enters a real data environment, the object that needs monitoring becomes the data itself. ChurnWatch covers the full chain with a very small pipeline: S3 for dataset management, SageMaker for model training, PSI for data drift detection, CloudWatch for metric monitoring.

The core conclusion can be distilled into a single sentence: the model can stay the same, but the data around it changes. Monitoring that change is essential work for keeping production machine learning systems healthy.

For developers just getting into MLOps, ChurnWatch offers a low-cost, reproducible starter template—no complex platform needed; a few AWS services plus a PSI script turn data drift from an abstract concept into visible metrics.

The typical MLOps lifecycle is usually divided into stages like data preparation, experimentation, training, deployment, monitoring, and retraining. ChurnWatch covers the training and monitoring ends while omitting heavyweight components like CI/CD, feature stores, and model registries. The value of this "minimal viable MLOps" approach is lowering the learning barrier: get the pipeline working first, then fill in the gaps as needed. To extend further, you could introduce Amazon SageMaker Model Monitor for automated drift detection, use SageMaker Pipelines to orchestrate retraining workflows, and use Feature Store to unify feature definitions—connecting "drift detected" to "automatically triggered retraining" and moving monitoring from observation toward self-healing.

One-line Summary

ChurnWatch builds a low-cost data drift monitoring template using S3, SageMaker, PSI, and CloudWatch.

Paragraph Summary

ChurnWatch is a small MLOps project built on the AWS ecosystem that demonstrates how to detect and monitor data drift in production. It uses S3 to manage training datasets, SageMaker to train a customer churn prediction model, and a Python script to compare distribution differences between baseline and simulated production data using the Population Stability Index (PSI), finally publishing drift values to CloudWatch as continuously observable time-series metrics. The project deliberately constructs production datasets with different distributions, giving the monitoring system real drift scenarios to face. The core conclusion: the model can stay the same, but the data around it keeps changing—training is just one step in the lifecycle, and once in production, the data itself is what needs continuous monitoring.