Eval Regression False Alarms? Compute a Noise Floor for Every Case First

A single eval run tells you almost nothing: one failed golden case doesn't mean much, and a passing run means even less. When the grader is string matching, a judge model, or pattern validation on free-text output, the same prompt can pass six times and fail twice with zero changes to the system.

The solution isn't a heavier eval framework — it's measuring a per-case noise floor from repeated baseline runs, plus one rule: when the data can't support a conclusion, refuse to escalate the verdict.

This article is based on a technical writeup about eval stability; the code reflects the original author's implementation approach.

Why One Run Can't Distinguish Noise from Regression

Think of each scored case as a coin, and you're trying to estimate its bias from a handful of flips. A prompt that passes 5 of 8 baseline repeats and 3 of 8 candidate repeats looks scary on a dashboard, but at n=8 the confidence intervals are so wide they can't support that alarm at all.

The math makes it obvious: 6 passes out of 8 gives a one-sided 95% Wilson lower bound of about 0.46; 3 out of 8 gives a lower bound of about 0.16. The candidate sits below the baseline lower bound, but it isn't zero — so the honest conclusion is "uncertain, need more samples," not "regression."

This is exactly where many eval suites quietly lose credibility. Teams wire thresholds to single runs, the threshold fires on sampling jitter, and within a month everyone has learned to ignore the report. A noise floor turns the report back into a signal, because it encodes how much the harness itself will fluctuate before any prompt change.

Wilson lower bound is a method for estimating confidence intervals on a binomial proportion (such as a pass rate), proposed by statistician Edwin B. Wilson in 1927. Compared to the common normal approximation (the Wald interval), the Wilson interval behaves more stably when sample sizes are small or proportions are near 0 or 1, and won't produce absurd bounds outside [0,1]. The core idea: the observed pass rate p̂ = k/n is just a noisy estimate of the true pass rate, which could be lower. A one-sided 95% Wilson lower bound gives a conservative limit — we're 95% confident the true pass rate is no lower than this value. In eval regression detection, we only have statistical grounds to say the candidate hasn't gotten worse if its lower bound exceeds the baseline's lower bound (or baseline point estimate); otherwise we can only call it "uncertain." This explains why 6 of 8 passes versus 3 of 8 passes can have wildly different point estimates yet overlapping intervals, making a regression conclusion impossible.

The Core Mechanism: Repeats, Lower Bounds, and Stamp Guards

The author built this gate in roughly eighty lines of standard-library Python, assuming you already store a stamp of model, sampling parameters, and grader version next to each score — because a score divorced from its stamp can't be compared to anything.

The module has three layers at its core: repeated runs, one-sided Wilson lower bounds, and quarantine for cases too unstable to judge at all. Two of these — collecting two baselines — are the part most people skip, yet it's exactly what's worth doing: both are generated by the same stamp against an unmodified prompt, so any gap between them is pure harness variance. That's precisely the quantity you need to know before blaming a prompt.

The exit code returns non-zero only when a case has zero passes, or still fails after escalation, so a noisy suite stops blocking deploys that never should have been blocked.

python eval_noise_floor.py collect --out runs/base_a.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py collect --out runs/base_b.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py collect --out runs/cand.json --repeats 8 \
  --model your-model-id --grader-version g7
python eval_noise_floor.py compare runs/base_a.json runs/base_b.json runs/cand.json

Repeated Runs: Cheap to Design, Expensive to Execute

Repeat counts scale cost linearly, and the arithmetic gets uncomfortable fast. Forty cases, k=8, two baselines plus one candidate is 960 generations; escalate every uncertain case to k=16 and you're past two thousand calls before you've tuned a single thing.

That cost structure is the main reason teams keep shipping single-run evals: the honest version is roughly ten times the traffic of the deceptive one.

The author notes that during the prototyping phase, some platforms offer free model access credits (such as promotional credits in the tens of millions of tokens) and optional free servers to run long repeat jobs. But these numbers and terms are promotional and time-limited, so confirm them on the project's official page before building a repeat budget around them. The author also made a disclosure: the article was prepared to promote a product.

Whichever endpoint fills the generate callable, keep the harness behind its own adapter. A regression gate that only runs on one provider's promotional credits isn't a gate — it's a dependency, and it dies the moment the terms change.

The independent and identically distributed (i.i.d.) assumption is the cornerstone of statistical inference for repeated runs. It requires that each repeat be independent of the others and follow the same probability distribution. In practice, this means: (1) each run must reinitialize the environment, with no state left over from the previous run; (2) sampling parameters (temperature, top-p, random seed, etc.) must be fixed throughout; (3) the grader version must be consistent, avoiding distribution drift from mid-stream upgrades. Once i.i.d. is violated (for example, a stateful agent modifies a database, or the previous output gets cached), the pass rate is no longer an unbiased estimate of the true pass rate, and the Wilson lower bound loses its meaning. That's why the article stresses "reruns must start from a reset fixture" — this isn't engineering fastidiousness, it's a precondition for statistical validity. Moreover, the choice of k=8 is a compromise between cost and statistical power: the larger k is, the narrower the confidence interval, but the number of generation calls also grows linearly.

What Each of the Four Verdicts Should Trigger

Verdict	Evidence Pattern	Action
within_noise	Candidate pass rate not below baseline lower bound	Ship, and record the pass rate so the lower bound drifts with reality
inconclusive	Below the lower bound, but at least one pass	Rerun only that case at k=16, then touch the prompt
regressed	Zero passes across all repeats, or still below the lower bound after k=16	Block the change, bisect the prompt or model stamp
quarantine	Two identical baseline runs differ by more than 0.25	Fix the grader first; this case can't serve as a gate

Quarantined Cases Are Often Misread as Model Problems

A judge model flipping on the same completion twice, or a whitespace-sensitive regex, both look like unstable prompts until you measure them twice. Pull those cases out of the gate, fix the grader, and reintroduce them with their own noise floor.

Grader noise refers to randomness introduced by the scoring step itself, not by the model output. Common sources include: a judge model (LLM-as-judge) giving different scores to the same output, regexes sensitive to whitespace or punctuation causing unstable matches, string matching sensitive to case or synonyms, and subjective disagreement in human labeling. When grader noise is high, the same model output can be judged pass or fail, inflating pass-rate variance. The article's signal of "two identical baseline runs differing by more than 0.25" is a sign of excessive grader noise — since both runs use exactly the same prompt and stamp, any gap can only be attributed to the grader itself. The right move here isn't to adjust the prompt or model, but to fix the grader first (e.g., switch to a more stable judge model, use deterministic scoring rules, or add repeated scoring with majority vote); otherwise that case can't serve as a reliable regression gate.

Limitations and When It Doesn't Apply

Scenarios using deterministic graders on structured output rarely need k=8, since exact key matching is essentially noise-free and one sanity repeat suffices.

Stateful agents that modify a database, queue, or sandbox between calls directly break the independence assumption — reruns must start from a reset fixture, or the lower bound is meaningless.

Teams that can't pin temperature, top-p, and grader version should stop here: repeats without a stable stamp just produce a more expensive guess.

In regulated environments with data retention obligations, read the terms before feeding golden cases into any free credits, and this harness offers no protection there. Finally, teams watching latency or cost budgets need a separate harness, because a pass-rate lower bound says nothing about the tail behavior you actually care about.

If a repeat-heavy version sounds worth trying and the token bill is the obstacle, free model access and free server options are a reasonable starting point. Keep the adapter boundary intact, verify the terms yourself, and the gate remains yours even after the promotion ends.

One-line Summary

Measure the noise floor with two baseline runs, then judge regressions with a Wilson lower bound to avoid single-run false alarms.

Paragraph Summary

This article argues that a single eval run can't distinguish sampling noise from real regression — at n=8 the confidence intervals are too wide. The solution is to measure a per-case noise floor from repeated baseline runs, then use one-sided Wilson lower bounds as the decision criterion. The core mechanism includes repeated runs, noise floor computation, and quarantining unstable cases. The four verdicts (within_noise/inconclusive/regressed/quarantine) map to different next actions, preventing false alarms from blocking deploys. Repeat costs scale linearly — forty cases at k=8 is about 960 generations, the main reason teams fall back to single-run evals. The article also covers inapplicable scenarios, such as stateful agents, teams that can't pin sampling parameters, and regulated environments.