Model Drift
A phenomenon in machine learning describing changes in model behavior over time. It is categorized into naturally occurring data distribution shift and deliberately introduced behavioral changes. The latter is more difficult to assess in terms of scope and predictability.
Core Facts
AI systems have introduced entirely new dimensions of security risk, including erroneous decisions caused by model hallucinations, prompt injection attacks, training data poisoning, and AI agent privilege abuse.
模型漂移通常分为数据漂移和概念漂移两类，数据漂移指输入数据统计分布随时间变化，概念漂移指输入与输出关系本身发生变化
Timeline (last 90 days)
Common practices for monitoring drift include statistically tracking changes in the distribution of input features (such as the PSI metric), tracking the distribution of online prediction results, and regularly sampling for manual review.
模型漂移指模型上线后由于真实世界数据分布变化导致预测性能随时间下降的现象，应对手段包括持续监控、性能告警阈值和定期或触发式再训练。
Covariate shift refers to the inconsistency of the feature distribution P(X) between source and target, while model shift involves the shift of the conditional distribution P(Y|X).
Covariate shift and model drift can trigger negative transfer, causing additional data to drag down model performance.
Factors influencing users' perception of model degradation include context window filling, system prompt changes, sampling temperature adjustments, and model drift.
Currently, both academia and industry lack standardized continuous monitoring benchmarks for models in production environments.
Testing for model drift requires establishing a continuous statistical monitoring pipeline, periodically comparing the distribution characteristics of production data and training data, and setting performance metric alert thresholds.
Performance degradation caused by model drift does not trigger system errors, making traditional monitoring methods (such as error logs and crash rates) completely ineffective against it.
作者记录一次无声的安全更新使管线准确率从96%骤降到71%，需要120个工程师工时诊断修复
All Facts (12)
AI systems have introduced entirely new dimensions of security risk, including erroneous decisions caused by model hallucinations, prompt injection attacks, training data poisoning, and AI agent privilege abuse.
80%Verified模型漂移通常分为数据漂移和概念漂移两类，数据漂移指输入数据统计分布随时间变化，概念漂移指输入与输出关系本身发生变化
65%UnverifiedAI product performance may deteriorate without any changes being made, due to reasons such as RAG knowledge base data not being synchronously updated or vendors silently updating model versions.
60%UnverifiedCommon practices for monitoring drift include statistically tracking changes in the distribution of input features (such as the PSI metric), tracking the distribution of online prediction results, and regularly sampling for manual review.
50%Unverified模型漂移指模型上线后由于真实世界数据分布变化导致预测性能随时间下降的现象，应对手段包括持续监控、性能告警阈值和定期或触发式再训练。
50%UnverifiedCovariate shift refers to the inconsistency of the feature distribution P(X) between source and target, while model shift involves the shift of the conditional distribution P(Y|X).
50%UnverifiedCovariate shift and model drift can trigger negative transfer, causing additional data to drag down model performance.
50%UnverifiedCurrently, both academia and industry lack standardized continuous monitoring benchmarks for models in production environments.
50%UnverifiedFactors influencing users' perception of model degradation include context window filling, system prompt changes, sampling temperature adjustments, and model drift.
50%UnverifiedPerformance degradation caused by model drift does not trigger system errors, making traditional monitoring methods (such as error logs and crash rates) completely ineffective against it.
50%UnverifiedTesting for model drift requires establishing a continuous statistical monitoring pipeline, periodically comparing the distribution characteristics of production data and training data, and setting performance metric alert thresholds.
50%Unverified作者记录一次无声的安全更新使管线准确率从96%骤降到71%，需要120个工程师工时诊断修复
50%