Wilson Lower Bound
A statistical method for estimating the confidence interval of a binomial proportion, proposed by Edwin B. Wilson in 1927. Compared with the normal approximation (Wald interval), it is more stable for small samples and proportions near 0 or 1, and does not produce bounds outside the range [0,1].
Timeline (last 90 days)
In repeated runs with n=8, the one-sided 95% Wilson lower bound for 6 passes out of 8 is about 0.46, and the lower bound for 3 passes out of 8 is about 0.16.
The Wilson lower bound is a method for estimating the confidence interval of a binomial proportion, proposed by statistician Edwin B. Wilson in 1927.
The independent and identically distributed (i.i.d.) assumption requires that each repeated run reinitialize the environment, fix sampling parameters, and keep the scorer version consistent; otherwise the pass rate is no longer an unbiased estimate and the Wilson lower bound loses meaning.
In evaluation regression detection, only if the candidate's lower bound is higher than the baseline's lower bound (or the baseline point estimate) is there statistical grounds to say it has not gotten worse; otherwise it can only be judged inconclusive.
All Facts (4)
In repeated runs with n=8, the one-sided 95% Wilson lower bound for 6 passes out of 8 is about 0.46, and the lower bound for 3 passes out of 8 is about 0.16.
50%UnverifiedThe Wilson lower bound is a method for estimating the confidence interval of a binomial proportion, proposed by statistician Edwin B. Wilson in 1927.
50%UnverifiedThe independent and identically distributed (i.i.d.) assumption requires that each repeated run reinitialize the environment, fix sampling parameters, and keep the scorer version consistent; otherwise the pass rate is no longer an unbiased estimate and the Wilson lower bound loses meaning.
50%UnverifiedIn evaluation regression detection, only if the candidate's lower bound is higher than the baseline's lower bound (or the baseline point estimate) is there statistical grounds to say it has not gotten worse; otherwise it can only be judged inconclusive.
50%