Statistical power
The probability that a statistical test correctly rejects the null hypothesis when the true difference equals or exceeds the equivalence margin; it is a key parameter in experimental design.
Timeline (last 90 days)
The study provides a closed-form formula that converts the equivalence margin and target power into the number of runs required for each method, with a shifted-t correction term in the formula calibrated to the actual two-level process.
To demonstrate equivalence at 0.01 AUROC with 80% power, twelve runs per method are required even if the two methods are exactly equal.
The experiment's statistical power is limited: a pair of designs needs a gap of roughly 6 to 9 percentage points to be judged significantly different.
The choice of k=8 is a compromise between cost and statistical power: the larger k is, the narrower the confidence interval, but the number of generation calls also grows linearly.
All Facts (4)
The study provides a closed-form formula that converts the equivalence margin and target power into the number of runs required for each method, with a shifted-t correction term in the formula calibrated to the actual two-level process.
50%UnverifiedTo demonstrate equivalence at 0.01 AUROC with 80% power, twelve runs per method are required even if the two methods are exactly equal.
50%UnverifiedThe experiment's statistical power is limited: a pair of designs needs a gap of roughly 6 to 9 percentage points to be judged significantly different.
50%UnverifiedThe choice of k=8 is a compromise between cost and statistical power: the larger k is, the narrower the confidence interval, but the number of generation calls also grows linearly.
50%