Three Runs Is Not a Sample Size: The Replication Fallacy in AI Evaluation and Its Computable Rule

A Question Everyone Skips

If you open a paper comparing two models, two agents, or two prompts at random and flip to the section describing how many experimental runs were performed, you'll almost always read the same sentence: "We ran three times," or "We used 5 random seeds." Then keep looking—find the sentence that says how much error control that number actually bought.

It isn't there.

This isn't carelessness; it's a missing measurement tool. The field spent a decade answering "how many items does one evaluation need": sequential stopping rules, power analysis, item response theory, variance decomposition. Those questions all have mature answers. But the replication axis (how many times you run the same configuration) has remained stuck on two folk constants—N=3 or N=5—with no derivation whatsoever. N=3 is a habit, not a calculation.

What's more uncomfortable is that the calculation takes only three lines.

There Is an Exactly Computable Rule for the Number of Replications

Suppose the probability that a single run reaches the wrong conclusion is p, and that the N runs are exchangeable (i.e., order doesn't affect the outcome). Then the probability that a strict majority vote over those N runs is still wrong is simply a binomial tail probability:

E(N, p) = P(Binomial(N, p) >= ceil(N/2))

No asymptotic approximation, no normality assumption, no "holds within some range" caveat. It uses the same arithmetic as the sign test—which is itself a hint about how ordinary this object is, and how rarely it's used here.

When p = 0.10 (a perfectly normal per-run disagreement rate for a stochastic evaluation), the error rates corresponding to those folk constants are:

Rule	N=3	N=5	N=10
Strict majority	0.02800	0.00856	0.000147
Unanimous	0.2710	—	0.6513
any-of (any pass)	0.0010	—	1.000e-10
Mean threshold	0.3085	—	0.05692

Unanimity: Adding Runs Makes It Worse

Two rows in this table deserve more attention than the familiar columns.

The first counterintuitive result: when unanimity is required, adding more runs makes things worse. At N=3 the error rate is 0.2710; at N=10 it actually rises to 0.6513. The reason is that requiring all runs to agree ANDs together N chances to fail: if any single run disagrees, the whole verdict fails. So "we only accept unanimous conclusions" is not a conservative option at all—past some N, it's worse than the complement of a coin flip. If your evaluation framework has a --require-unanimous flag, it has a cost, and nobody has ever quoted it.

The any-of rule (any pass counts as a pass) sits at the other extreme in this table: at p=0.10 and N=3 the error rate is only 0.0010, dropping to 1.000e-10 at N=10. The intuition is that as long as one run avoids the error, the verdict stands, so an error requires all N runs to fail simultaneously—a probability on the order of p to the Nth power, collapsing geometrically in N. This is the mirror image of unanimity: unanimity requires avoiding any single failure, accumulating the complement of 1; any-of requires avoiding total failure, decaying as a power of p. Both differ from strict majority voting, whose error is a binomial tail probability that falls at a rate between the two. Choosing a rule is essentially choosing where errors enter: majority spreads error across all the roughly half-and-half vote patterns, any-of squashes it into the rare event of total wipeout, and unanimity amplifies it into the increasingly common event of any single disagreement. These three curves never intersect at the same p, but across different p ranges the optimal rule quietly changes hands.

The Mean Score Is a Different Rule, Not a Safer One

The second conclusion: switching from majority voting to a mean threshold swaps in an entirely different mathematical regime. any-of collapses geometrically (0.0010 → 1.000e-10, a drop of ten orders of magnitude over one span), whereas the mean-threshold rule falls only fivefold (0.3085 → 0.05692). People switch from "majority vote" to "mean score" because they believe the latter is "more robust"—but that switch changes the entire error mechanism you're relying on.

One more detail worth noting: at equal compute cost, an even N beats the odd N just above it. At p=0.10, N=3 has an error rate of 0.0280, N=4 is 0.00370, and N=5 is 0.00856. This upgrades "how do we handle ties" from an implementation detail to a design decision with a clear price tag—"we count ties as failures" is a policy, and now you can calculate what it costs.

Exchangeability is the only structural assumption here, and it deserves separate mention. It means that shuffling the order of the N runs arbitrarily leaves the joint distribution unchanged—in other words, there's no systematic difference between the first and second run, and the run index carries no information. This is weaker than "independent and identically distributed": i.i.d. implies exchangeable, but exchangeable allows correlation across runs (e.g., sharing a random seed pool, a machine, or a day's data snapshot). It's precisely because only exchangeability is required that the binomial tail probability holds without assuming independence—the misjudgment rate of the vote depends only on "each run having the same marginal error probability, and no run being systematically singled out." Conversely, if an order effect exists (say, the first run is systematically slower due to a cold cache), exchangeability breaks and the closed-form solution fails with it. So in practice, randomizing run order, or resampling seeds for every configuration, is a necessary precondition for making this rule usable.

Two Needs, Intersecting at a Measurable Point

There are two reasons to run more replications, and they point in opposite directions:

• Estimation need: you want to know p, the rate at which a single run disagrees with itself. More replications narrow it, and the cost is low when p is small.
• Decision need: you want the majority vote verdict itself to be correct. This corresponds to the tail probability above, and it becomes sharply more expensive as p approaches 1/2—at which point a single run is close to a coin flip, and a majority vote of many coin flips isn't much better.

These two curves intersect. In the study underlying this post, the intersection was located by grid search: at p = 0.35 the estimation need dominates, at p = 0.40 the decision need takes over, so p* = 0.40—corroborated by the adjacent cells on either side rather than fitted. Below the intersection, running more times to measure p buys you what you actually need; above it, no amount of measurement will save you, because the decision itself is the binding cost.

This yields a usable rule: before deciding what "run more times" means, check which side of p* your measured p falls on.

When Items and Replications Compete for the Same Budget

If a fixed compute budget can buy either more items or more replications, the optimum is not a matter of taste. Let the per-item variance be sigma^2 and the per-replication (resampling) variance be tau^2; then the boundary is:

N* = sqrt( a * sigma^2 / (b * tau^2) )

It is scale-free in the budget, and the corresponding AM-GM optimum is Var* · B = (sqrt(a*tau^2) + sqrt(b*sigma^2))^2. Two consequences worth remembering:

• This boundary is a ratio of variances, not a preference. If you can't state tau^2—the variance of the same configuration across replications—you can't state how many replications to run. This quantity is measurable, yet almost nobody reports it.
• Integrality at the boundary matters. Once N is explicitly integerized, in the interval where the continuous solution falls between 1 and 2 runs, the optimum shifts from N*=1 to N*=sqrt(2)—which is precisely the budget range where people argue endlessly about "one run or two."

There is a third knob: the difficulty/scale T of the items. Across a 128x span, the two input terms behave as tau^2(T) = c_tau / T and sigma^2(T) = sigma_inf^2 + c_sig / T, yielding an interior three-knob optimum (T*, N*) rather than a corner solution.

The formula N* = sqrt( a * sigma^2 / (b * tau^2) ) comes from a classic optimal allocation problem: a total budget B is split into a·N (replication-side cost) and b·T (item-side cost), and the final estimator's total variance takes the form sigma^2 / (N·T) + tau^2 / N, whose Lagrangian optimization gives this square-root rule. Its shape is itself the conclusion: the optimal number of replications depends only on the ratio of the two variances, not on the absolute size of the budget—doubling the budget raises both item count and replication count while leaving their ratio unchanged. This explains a common confusion: why "our machines are stronger now, so let's run more times" is never a sufficient argument unless tau^2/sigma^2 itself has changed. The AM-GM line gives the minimum achievable variance-times-budget product under this constraint; it's a lower bound that no allocation scheme can beat. The integrality point is a reminder that when the continuous solution falls between 1 and 2, rounding jumps to 1 or 2, and the actual trade-off those two integers represent ("run once and gamble" versus "run twice and average") is exactly the range most argued about in practice and most easily led astray by intuition.

A Result That Overturns Conventional Wisdom

The following result deserves to be pinned on the wall. Across a range spanning 256x in item count, the single-run misjudgment rate barely moves:

• At T=16 it's 0.531
• At T=4096 it's 0.534

When two models are nearly tied, buying more items does not make a single-run verdict reliable. No matter how large the test set, the single-run error rate stays at roughly one-half. The reason is that what varies across runs is not the kind of measurement noise that averaging over more items can cancel out. "A larger test set" and "more random seeds" are not substitutes; on this axis, they aren't even in the same conversation.

Both numbers above come from the study's own measurement tooling, and this shape is not an artifact of a Gaussian data-generating process: on a fixed-version public benchmark (UCI Concrete Compressive Strength, sha256 d7d8bd08…), tau^2(T) decays as 1/T with a log-log slope of -0.961, matching the theoretical prediction of -1.

Measure, Don't Assume

This rule is only useful when its inputs are measurable, so the study chose to measure rather than assume. On a synthetic stochastic optimization task, the inputs measured as sigma^2 = 4.930063e-04 and tau^2 = 2.044068e-04. In real stochastic optimization runs, the measured p spanned from 0.0065 all the way to 0.5790—a 90x spread across five cells, with four of them above 0.10.

That spread is itself the argument for the whole endeavor. When p = 0.0065, three runs is already generous and the honest move is to spend the money on items; when p = 0.5790, three runs is nearly useless, and no reasonable N can rescue a decision built on that comparison. The same folk constant is right in one cell and wrong by an order of magnitude in another—and the only thing distinguishing them is a number nobody measured.

What You Can Do Starting Monday

1. Measure p—the rate at which a single run reaches a different conclusion. If a paper doesn't report this number, it hasn't made the claim it appears to make.
2. Choose the aggregation rule deliberately, including tie handling, and state what "a tie" means.
3. Compute N with this rule rather than inheriting it; if items and replications are competing for the same budget, report tau^2.

Reproducing

The study is released as a package that runs end to end:

bash reproduce.sh          # Full: rebuild, recompute, plot, validate (~95 seconds)
REPRODUCE_QUICK=1 bash reproduce.sh   # Validate committed artifacts (~25 seconds)

The verifier's checks are listed in the README and printed at runtime: VALIDATE 46/46, RESULT: PASS, and both planted controls ALL PLANTS CAUGHT. The 46 here is an integer count (one check per claim, each tied to a specific claim in the manuscript), not a statistical threshold. The package's own two-way contract is this: every file it regenerates is byte-identical on a second run (37 committed files, verified with sha256; only run logs are excluded, as noted in the README). I ran it twice locally to confirm this rather than just reading the documentation: results matched file by file.

The exact arm is cross-validated by two independent computation paths, agreeing to 1.447e-13 across 147 cells, plus a 200,000-trial Monte Carlo. The reference layer contains 108 citation entries (the bar is 100).

A boundary note, because it constrains every number above. The empirical arm is a controlled instrument: a synthetic stochastic optimization plus a fixed-version public benchmark, not a census of published papers. The flat-p counterexample, the 90x spread, and the -0.961 slope are properties of those measurements; but the arithmetic is not—the binomial tail probability holds for any p you can measure. Every number in the paper is parsed from committed artifacts by the build script that renders it; the build refuses to run when it detects drift, and it carries its own planted controls—that's the kind of "reproducible" worth demanding from journals.

The full paper is published in SILICON SCIENCE: Computer Science, an AI-operated peer-reviewed journal: submissions arrive as issues, reviews are public, and every published paper ships with a reproduction script and data statement. Paper link: How Many Runs Does a Claim Need?