Data Drift

Drift detection involves a trade-off: setting the threshold too sensitively leads to frequent false alarms and unnecessary retraining, while setting it too loosely may allow model performance to continuously degrade without being detected.

Data Drift refers to the continuous degradation of model performance without any errors when the statistical properties of real online data deviate from the training set.

Data drift or concept drift refers to the phenomenon where model performance degrades significantly when the real-world data distribution changes while the model is still trained on old data; the industry addresses this with continuous learning and active learning strategies.

模型漂移通常分为数据漂移和概念漂移两类，数据漂移指输入数据统计分布随时间变化，概念漂移指输入与输出关系本身发生变化

The core challenge of MLOps lies in model drift: as real-world data distributions change over time, model prediction accuracy gradually declines.

Performance drift refers to the phenomenon of gradual degradation in model output quality over time, which may stem from silent updates by the model provider, changes in the distribution of contextual data, and updates to the knowledge base of the RAG system.

State-drift is different from data drift or concept drift, which focus on changes in the model's input distribution or input-output mapping and can be detected with observability tools; state-drift concerns the expiration of stored beliefs, which can fail even when the input is entirely in-distribution.

Concept drift refers to the mapping relationship between input data and target labels changing over time, which is different from data drift where only the input distribution changes.

Data drift does not trigger obvious errors; it only manifests slowly in business metrics.

It is necessary to regularly monitor the confidence distribution in production environments to prevent data drift from causing a previously well-calibrated model to become miscalibrated on new data.

When the distribution of online data deviates from the training data, model performance degrades (data drift and model decay), requiring retraining to be triggered.

The results of human review should flow back to continuously calibrate the model and costs; otherwise the system will fail as data drifts.

Production environment data changes dynamically, as new programming languages, key formats, and code patterns continuously emerge, causing data drift

Data drift and concept drift can cause model performance degradation, but may show no anomalies in service-layer health metrics (latency, throughput, error rate), requiring active detection through statistical methods such as KL divergence, PSI, and KS tests.

Commercial data science projects must face challenges that are almost never encountered in personal projects, such as data drift, model interpretability requirements, A/B testing and causal inference, and full lifecycle MLOps management.

数据漂移（Data Drift）指生产环境中输入数据的统计分布随时间发生变化，导致模型接收数据与训练时分布不匹配