Don't Just Look at Averages When Switching Models: Use Paired Testing to Catch Hidden Regressions

Two models both score 71% on the same fixed test set, and the switch looks safe—until you compare case by case and find 9 cases got better, 9 got worse, and users will only notice the 9 that got worse. Aggregate scores hide the flip cases that actually break your product.

The solution isn't a bigger benchmark. It's paired testing with a frozen gold set, and auditing your grader before you trust it. Below we build the whole harness from start to finish: freezing cases, auditing the grader, running and reporting, and the limits where this approach breaks down.

Freeze the cases: lock the gold set before the experiment starts

A gold set is only meaningful if it's immutable during the experiment. The approach: write cases to disk once, hash the file, and record that hash in every report. Each case needs three fields: a stable identifier, the exact prompt to send, and a deterministic checker that returns a boolean.

Keep checkers deliberately "boring": JSON Schema validation, numeric comparison with tolerance, required-substring checks. These judgments won't drift because the underlying judge model changed. Tag each case with a capability label—extraction, refusal, format—so later regressions can be localized instead of argued about.

The first 80 cases are a reasonable floor, and that's not arbitrary: the statistical power of a paired test comes only from discordant pairs. If both models pass every case, a bigger set gives you zero information. Curate deliberately toward cases near the decision boundary—the ones you suspect a cheaper or differently tuned model will get wrong. If a case has never failed on any backend you've tried, it's a smoke test, not a regression test, and it belongs in a different file.

Audit the grader first with canaries

Before running any comparison, feed the grader several outputs you know are wrong and require it to reject all of them. This is cheap insurance against the most embarrassing failure mode in evaluation work: someone loosened a regex and the grader quietly started passing everything. Keep the canary file alongside the gold set and version them together. A canary surviving a grader change is a stop-the-line event, not a warning.

# graders.py — deterministic checkers + canary audit
import json, re

def check_json_keys(keys):
    def _check(output: str) -> bool:
        try:
            obj = json.loads(output)
        except json.JSONDecodeError:
            return False
        return all(k in obj for k in keys)
    return _check

def check_number(expected: float, tol: float = 1e-6):
    def _check(output: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", output)
        return bool(m) and abs(float(m.group()) - expected) <= tol
    return _check

CANARIES = [
    ("empty_json",  check_json_keys(["id", "total"]), "{}"),
    ("prose_only",  check_json_keys(["id", "total"]), "Sure! Here is the answer."),
    ("wrong_total", check_number(41.50), "The total is 14.50"),
    ("near_miss",   check_number(41.50), "41.5 apples"),  # Must pass: tolerance case
]

def audit(canaries):
    must_fail = [c for c in canaries if c[0] != "near_miss"]
    survived = [name for name, check, out in must_fail if check(out)]
    if survived:
        raise SystemExit(f"grader audit failed, canaries passed: {survived}")
    for name, check, out in canaries:
        if name == "near_miss" and not check(out):
            raise SystemExit("grader too strict: tolerance case rejected")

The last canary matters as much as the first three: a grader that rejects everything will invent a regression story out of thin air. Auditing in both directions keeps the check honest, and it's about ten minutes of work once per project.

Run both legs and record the manifest

The harness needs to send the same frozen prompts to both backends and persist the raw outputs. Re-running a flaky endpoint mid-analysis silently destroys your paired design. Cache each response keyed by case identifier, prompt text, and backend label, and write a manifest containing the backend identifier string, sampling parameters, and gold-set hash.

If you can't state the exact backend version that served a request, you don't have a reproducible comparison—you have a timestamped anecdote.

**Paired design** is the statistical foundation here: the same case goes to both models, and what's compared is the pass/fail status on that same case, not the pass rates of two independent samples. Each case serves as its own control, eliminating variance from differences in case difficulty. Under a paired design, only discordant pairs—where one model passes and the other fails—carry information. Cases where both pass or both fail contribute nothing to delta. So no matter how large the sample, if every pair is concordant, the test has zero power. In practice this requires each case to be judged exactly once under a fixed backend (or k repetitions with majority vote), with both legs using exactly the same prompt. Any routing jitter, temperature sampling, or re-running of unstable endpoints degrades paired data into unpaired data and invalidates the McNemar test's assumptions.
# swap_test.py — paired comparison + exact McNemar test
import json, hashlib, math, os, urllib.request
from collections import Counter

def sha(*parts: str) -> str:
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]

def call(endpoint: str, model_tag: str, prompt: str) -> str:
    body = json.dumps({"model": model_tag, "temperature": 0,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(endpoint + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['API_KEY']}"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

def exact_mcnemar(b: int, c: int) -> float:
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(0, min(b, c) + 1)) / 2 ** n
    return min(1.0, 2 * tail)

def run(cases, backends, cache_path="cache.jsonl"):
    cache = {}
    if os.path.exists(cache_path):
        for line in open(cache_path):
            row = json.loads(line)
            cache[row["key"]] = row
    for case in cases:
        for name, (endpoint, tag) in backends.items():
            key = sha(name, case["id"], case["prompt"], tag)
            if key not in cache:
                out = call(endpoint, tag, case["prompt"])
                cache[key] = {"key": key, "case": case["id"], "backend": name,
                              "tag": tag, "output": out,
                              "passed": bool(case["check"](out))}
                with open(cache_path, "a") as f:
                    f.write(json.dumps(cache[key]) + "\n")
    return cache

def report(cases, cache, old, new):
    pairs = []
    for case in cases:
        a = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == old)
        b = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == new)
        pairs.append((case["id"], case.get("tag", "untagged"), a["passed"], b["passed"]))
    b_only = sum(1 for _, _, p, q in pairs if q and not p)   # new passes, old fails
    c_only = sum(1 for _, _, p, q in pairs if p and not q)   # old passes, new fails
    delta = (b_only - c_only) / len(pairs)
    flips = [(cid, tag) for cid, tag, p, q in pairs if p and not q]
    return {"n": len(pairs), "old_pass": sum(p for _, _, p, _ in pairs),
            "new_pass": sum(q for _, _, _, q in pairs), "gain": b_only,
            "loss": c_only, "delta": round(delta, 4),
            "p_value": round(exact_mcnemar(b_only, c_only), 4),
            "flips_by_tag": Counter(t for _, t in flips),
            "flip_list": [cid for cid, _ in flips]}

cache_backend_tag is a two-line lookup against the cache; pulling it out keeps the report function readable. Note that delta is computed only from discordant pair counts—which is exactly why two runs can have unchanged averages while a real regression exists.

Read the report as decisions, not scores

The output is a small dict where each field maps to an action. Example: {"n": 96, "gain": 7, "loss": 11, "delta": -0.0417, "p_value": 0.48, "flips_by_tag": {"format": 6, "extraction": 5}}. These numbers are a template, not results from any real system.

Observation	Interpretation	Action
Canary audit fails	The grader is broken	Stop; fix the grader before any comparison
delta ≥ 0 and high p-value	No regression detected	Keep the switch, keep the gold set
delta < 0, loss > gain, p < 0.05	Real regression on this set	Block the switch or narrow the prompt scope
delta < 0, p > 0.2	Insufficient power, not necessarily fine	Add boundary cases before deciding
Flips concentrated in one tag	Localized breakage	Manually inspect flip_list, typically 6–12 cases

The concentration column carries most of the practical value: 11 losses spread evenly across five capabilities looks like noise, while 11 losses all inside format usually points to a single missing instruction. Read the flip list before the p-value; read the raw outputs before the flip list.

The exact McNemar test compares binary outcomes for the same set of subjects under two conditions. It looks only at the counts in the two discordant directions: b = old fails and new passes, c = old passes and new fails. Under the null hypothesis, b and c follow a binomial distribution with n = b + c and p = 0.5. The exact version computes the two-sided tail probability of that binomial directly, without relying on the large-sample chi-square approximation, so it remains reliable when b + c is small (18 in this example). The p-value answers: "Could the observed flip imbalance plausibly arise from random fluctuation alone?" A high p-value doesn't mean the models are equivalent—only that the current data isn't enough to reject "no difference." That's exactly why the table calls for more boundary cases when delta < 0 and p > 0.2: with insufficient power, not significant doesn't mean safe.

Where this harness breaks down

The paired design assumes each case is a Bernoulli trial under a fixed backend, so a backend that routes consecutive requests to different model versions inflates variance and silently breaks the pairing. Shared free tiers are the highest risk, which is why the manifest records the returned identifier string verbatim rather than trusting the name you requested. If you sample at any temperature above zero, run k repetitions per case and take a majority vote for the case-level verdict, then run the canary audit again on the majority verdicts.

This is also not a load test: it covers nothing about tokens per second, queueing behavior, or cost at scale—those need separate methods. Teams needing regulator-grade evaluation evidence, or products whose outputs have no deterministic anchor at all, should hand this work to human review rather than stretching boolean checkers past what they can support. Nor does it replace product-level tracing, because a gold set only knows the failures you've already imagined.

If you don't want to pay for two full backend bills, MonkeyCode's free model access and free server options are enough to run the harness above on a small set. (Disclosure: this article was written as part of a MonkeyCode product promotion.) Start with twenty boundary cases, one canary file, and one fixed backend label per leg, and let the flip list tell you whether the switch is worth the rest of your afternoon.

The Bernoulli trial assumption means each case's pass/fail is an independent, identically distributed binary outcome, with the pass probability determined by a fixed backend version. The paired McNemar test rests on this. If a backend routes consecutive requests to different model versions, the pass probability for the same case is no longer fixed—it's equivalent to mixing several different Bernoulli distributions, variance is artificially inflated, and discordant pair counts lose their clear meaning. Shared free tiers, canary releases, and automatic model upgrades all introduce this risk. The mitigation is to record the model identifier string returned by the response verbatim in the manifest (not the name you filled in the request), and verify before analysis that this string stayed consistent throughout the run; if not, re-pair segmented by version, or pin routing to a single version and run again.

One-line Summary

Use a frozen gold set and the paired McNemar test to expose model-switch regressions hidden by aggregate scores.

Paragraph Summary

The article argues that two models scoring the same aggregate (say, 71%) on a fixed test set doesn't mean the switch is safe, because case-by-case comparison may reveal equal numbers of improved and worsened cases—and users only perceive the worse ones. The fix isn't a bigger benchmark but a paired-testing harness: freeze and hash the gold set before the experiment, give each case a deterministic checker; audit the grader bidirectionally with canaries so it both rejects wrong outputs and accepts tolerance cases; send identical prompts to both backends, cache raw outputs, and record backend identifiers and parameters; compute delta and p-values via the exact McNemar test on discordant pairs, and track flip concentration by capability tag. Read the report as decisions: a failed canary is a stop-the-line event, delta < 0 with p < 0.05 blocks the switch, and flips concentrated in one tag warrant manual inspection of the flip list first. Finally, the failure boundaries: backend routing jitter breaks the paired assumption, the harness doesn't cover load testing or product-level tracing, and scenarios lacking deterministic anchors should go to human review.