Model Behavior Drift
Model behavior drift refers to the phenomenon in which the output distribution for the same input changes even when code and prompts are unchanged, due to silent server-side model updates, changes in temperature parameters, or adjustments to context length. It arises from the random sampling mechanism of model inference and vendors' continuous iteration on model versions.
Timeline (last 90 days)
Industry approaches to coping with model behavior drift include: vendors providing dated fixed version snapshots, the application side maintaining golden test sets for regression, and incorporating prompts/model version numbers/evaluation results into version management.
Model behavior drift means that even if the code and prompts are unchanged, after the underlying model is silently updated, temperature parameters change, or context length is adjusted, the output distribution for the same input may also change.
All Facts (2)
Industry approaches to coping with model behavior drift include: vendors providing dated fixed version snapshots, the application side maintaining golden test sets for regression, and incorporating prompts/model version numbers/evaluation results into version management.
50%UnverifiedModel behavior drift means that even if the code and prompts are unchanged, after the underlying model is silently updated, temperature parameters change, or context length is adjusted, the output distribution for the same input may also change.
50%