Model Swaps: Don't Just Look at the Average Score—Use Paired Tests to Detect Real Regressions

Title

Model Swaps: Don't Just Look at the Average Score—Use Paired Tests to Detect Real Regressions

Meta Description

Identical average scores can hide real regressions when swapping models. This article shows how a frozen golden set, canary-audited graders, and an exact McNemar paired test detect per-case flips during model switches, preventing aggregate metrics from masking product-level breakage.

Article Content

Two backends both score 71% on the same frozen test set—sounds safe to switch, right? But a case-by-case comparison reveals nine cases got worse and nine got better—and users will only notice the nine that got worse. This isn't a problem solved by a bigger benchmark. It calls for a paired test against a frozen golden set, paired with an audited grader. This article builds that test harness from scratch and explains how to interpret its output correctly.

Why average scores mask real regressions

Aggregate metrics hide the flips that actually break the product. Two backends can have the same average score on the same frozen set while their per-case pass/fail outcomes are completely opposite. A paired test focuses only on discordant pairs, so it can detect a statistically significant regression even when the two runs have identical average scores.

The core of this approach isn't scaling up the benchmark—it's increasing the discriminating power of the test set. If two models pass every case, a larger test set provides no information. When selecting cases, prioritize examples near the decision boundary where you suspect a cheaper or differently tuned model will fail. Cases that never fail are just smoke tests, not regression tests, and should live in a separate file.

Step 1: Freeze the cases before touching any model

A golden set is only valuable if it stays immutable during the experiment. Write the cases to disk once, hash the file, and record that hash in every report. Each case needs three fields: a stable identifier, the exact prompt to send, and a deterministic checker that returns a boolean.

Keep checkers deliberately boring—JSON Schema validation, numeric comparisons with tolerance, required-substring checks—so verdicts don't drift when the judge model changes. Tag each case with a capability label (e.g., extraction, refusal, format) so you can later locate which category degraded instead of arguing about where things went wrong.

Eighty cases is a reasonable minimum for a first round: the power of a paired test comes from discordant pairs, and a set where everything passes is useless no matter how big it is.

Step 2: Audit the grader with canaries first

Before any comparison run, feed the grader a few outputs you know are wrong and require it to reject all of them. This is cheap insurance against the most embarrassing failure mode in evaluation work: someone loosens a regex and the grader quietly starts passing everything.

Keep the canary file alongside the golden set and version them together. A canary surviving a grader change should be treated as a stop-the-line event, not a warning.

# graders.py — deterministic checkers plus canary audit
import json, re

def check_json_keys(keys):
    def _check(output: str) -> bool:
        try:
            obj = json.loads(output)
        except json.JSONDecodeError:
            return False
        return all(k in obj for k in keys)
    return _check

def check_number(expected: float, tol: float = 1e-6):
    def _check(output: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", output)
        return bool(m) and abs(float(m.group()) - expected) <= tol
    return _check

CANARIES = [
    ("empty_json",  check_json_keys(["id", "total"]), "{}"),
    ("prose_only",  check_json_keys(["id", "total"]), "Sure! Here is the answer."),
    ("wrong_total", check_number(41.50), "The total is 14.50"),
    ("near_miss",   check_number(41.50), "41.5 apples"),  # must pass: tolerance case
]

def audit(canaries):
    must_fail = [c for c in canaries if c[0] != "near_miss"]
    survived = [name for name, check, out in must_fail if check(out)]
    if survived:
        raise SystemExit(f"Grader audit failed, canaries passed: {survived}")
    for name, check, out in canaries:
        if name == "near_miss" and not check(out):
            raise SystemExit("Grader too strict: tolerance case rejected")

The last canary is as important as the first three: a grader that rejects everything will happily manufacture a regression story for you. Auditing in both directions keeps the check honest, and it takes about ten minutes to write once per project.

Step 3: Run both legs and record a manifest

The harness needs to send the same frozen prompts to both backends and persist the raw outputs—re-running unstable endpoints mid-analysis silently invalidates the paired design. Cache each response keyed by a hash of the case identifier, prompt text, and backend tag, and write a manifest containing the backend identity string, sampling parameters, and golden-set hash.

If you can't name the exact backend version that served a request, you don't have a reproducible comparison; you have a timestamped anecdote.

**Exact McNemar test** compares binary outcomes from two related samples, which fits this scenario's paired pass/fail results for the same case on two backends. It uses only discordant pairs: the number of gains `b` (new passes, old fails) and losses `c` (old passes, new fails). Under the null hypothesis "both backends have the same pass rate," `b` and `c` follow a binomial distribution with success probability 0.5. The exact version computes binomial tail probabilities directly rather than using the large-sample chi-squared approximation, so it still gives reliable p-values when `b + c` is small. The `exact_mcnemar(b, c)` function in the code takes twice the one-sided tail probability as the two-sided p-value and clamps it to 1.0. When `b + c` is zero it returns 1.0, meaning no evidence.
# swap_test.py — paired comparison with exact McNemar test
import json, hashlib, math, os, urllib.request
from collections import Counter

def sha(*parts: str) -> str:
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]

def call(endpoint: str, model_tag: str, prompt: str) -> str:
    body = json.dumps({"model": model_tag, "temperature": 0,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(endpoint + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['API_KEY']}"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["choices"][0]["message"]["content"]

def exact_mcnemar(b: int, c: int) -> float:
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(0, min(b, c) + 1)) / 2 ** n
    return min(1.0, 2 * tail)

def run(cases, backends, cache_path="cache.jsonl"):
    cache = {}
    if os.path.exists(cache_path):
        for line in open(cache_path):
            row = json.loads(line)
            cache[row["key"]] = row
    for case in cases:
        for name, (endpoint, tag) in backends.items():
            key = sha(name, case["id"], case["prompt"], tag)
            if key not in cache:
                out = call(endpoint, tag, case["prompt"])
                cache[key] = {"key": key, "case": case["id"], "backend": name,
                              "tag": tag, "output": out,
                              "passed": bool(case["check"](out))}
                with open(cache_path, "a") as f:
                    f.write(json.dumps(cache[key]) + "\n")
    return cache

def report(cases, cache, old, new):
    pairs = []
    for case in cases:
        a = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == old)
        b = next(r for r in cache.values() if r["case"] == case["id"] and r["backend"] == new)
        pairs.append((case["id"], case.get("tag", "untagged"), a["passed"], b["passed"]))
    b_only = sum(1 for _, _, p, q in pairs if q and not p)   # new passes, old fails
    c_only = sum(1 for _, _, p, q in pairs if p and not q)   # old passes, new fails
    delta = (b_only - c_only) / len(pairs)
    flips = [(cid, tag) for cid, tag, p, q in pairs if p and not q]
    return {"n": len(pairs), "old_pass": sum(p for _, _, p, _ in pairs),
            "new_pass": sum(q for _, _, _, q in pairs), "gain": b_only,
            "loss": c_only, "delta": round(delta, 4),
            "p_value": round(exact_mcnemar(b_only, c_only), 4),
            "flips_by_tag": Counter(t for _, t in flips),
            "flip_list": [cid for cid, _ in flips]}

Note that delta is computed solely from discordant counts, so an unchanged average score across two runs can absolutely coexist with a real regression.

Step 4: Read the report as a decision, not a score

The output is a small dictionary, and each field maps to an action. A sample shape like {"n": 96, "gain": 7, "loss": 11, "delta": -0.0417, "p_value": 0.48, "flips_by_tag": {"format": 6, "extraction": 5}}—these numbers are a template, not the result of anyone's system.

Observation	Interpretation	Action
Canary audit fails	Grader is broken	Stop; fix the grader first
delta ≥ 0 and high p-value	No regression detected	Keep the swap, keep the golden set
delta < 0, loss > gain, p < 0.05	Real regression on this set	Block the swap or narrow the prompt scope
delta < 0, p > 0.2	Underpowered, not necessarily fine	Add boundary cases before deciding
Flips concentrated in one tag	Localized breakage	Manually inspect the flip list

The concentration column carries most of the practical value: eleven losses spread evenly across five capability categories looks like noise, while eleven losses concentrated in format usually points to a missing instruction. Read the flip list before the p-value; read the raw outputs before the flip list.

Statistical power is the probability that a test detects a regression when one truly exists. The power of the paired McNemar test comes almost entirely from the number of discordant pairs (b+c), not the total number of cases n. If only 5 out of 80 cases flip, the p-value may be far above 0.05 even when a real effect exists, because there isn't enough information. A p > 0.2 should be read as "insufficient evidence to conclude," not "no regression." To increase power, add cases near the decision boundary that are likely to flip, rather than simply enlarging the set of cases that all pass.

Where this harness stops

The paired design assumes each case is a Bernoulli trial under a fixed backend, so a backend that routes consecutive requests to different model versions inflates variance and silently breaks the pairing. Shared free tiers are the highest risk, which is exactly why the manifest records the returned identity string verbatim rather than trusting the name you requested. If sampling at any temperature above zero, run k repetitions per case, decide the case-level verdict by majority, then re-run the canary audit on the majority verdicts.

This also isn't a load test—it says nothing about tokens per second, queueing behavior, or cost at scale, which require a separate methodology. Teams needing regulator-grade evaluation evidence, or products whose outputs have no deterministic anchor at all, should put this work in front of human review rather than stretching boolean checkers past what they can support. Nor does it replace product-level tracing, because a golden set only knows the failures you've already imagined.

If you want to run the reference leg without paying for two full backend bills, MonkeyCode's free model access and free server option are enough to execute the harness above on a small set. Disclosure: this article is part of a MonkeyCode product promotion. Start with twenty boundary cases, one canary file, and one fixed backend tag per leg, and let the flip list tell you whether the swap is worth the rest of your afternoon.

The Bernoulli trial assumption requires that each case's pass/fail under a fixed backend and fixed sampling parameters is an independent binary outcome. If a backend routes consecutive requests to different model versions, multiple calls for the same case may yield different results, making the case-level verdict unstable; the paired design assumes results for the same case on two backends can be compared pairwise, and routing changes introduce extra variance. Sampling at temperatures above zero breaks the assumption in the same way, which is why the article recommends running k repetitions per case, deciding the case-level verdict by majority, and then re-auditing canaries on the majority verdicts to restore comparability.

One-line Summary

Paired testing and the McNemar test reveal model-swap regressions hidden by average scores.

Paragraph Summary

The article notes that two backends can have identical average scores on a frozen test set, yet a case-by-case comparison may show nine worse and nine better—users only notice the worse ones. It proposes a paired test harness: first freeze the golden set and record its hash, giving each case a deterministic checker and capability label; then audit the grader with two-way canaries that both reject known-wrong outputs and accept tolerance boundaries; then run both legs and cache raw outputs, using an exact McNemar test to compute p-values and delta from discordant pairs; finally read the report as a decision, using flip tags to localize problems. The article also notes the limits of the paired design, such as backend routing and sampling temperature breaking assumptions, and that it cannot replace load testing or product-level tracing.