Concept
Sampling Bias
A phenomenon in which the composition of an evaluation set deviates from the real deployment distribution because of human selection, causing measured scores to be systematically distorted. It commonly occurs when the purposes of training and evaluation are conflated.
Timeline (last 90 days)
Oct 26
In the evaluation context, sampling bias refers to the composition of an evaluation set deviating from the real deployment distribution due to human selection, causing systematic distortion of scores; practices to avoid it include freezing the evaluation set before training, stratified sampling according to the real distribution, and retaining a holdout set that never participates in tuning.
Unverified50%