Two Runs, One Verdict: How to Verify That an AI Coding Patch Reproduces

When the Team Can't Stop Arguing About the Same Bug

A worker retries a failed task three times and occasionally duplicates a side effect. In the incident thread, nobody disagrees about the symptom, but everyone disagrees about the cause, and the argument has outlasted the bug it was supposed to fix. Someone suggests letting an AI coding assistant write the patch, and the thread splits again: half want the diff right now, the other half refuse to trust a patch that no one can reproduce.

That split isn't a debate about model quality, and no further opinions will settle it. What settles it is evidence narrow enough to produce inside a single meeting block: two attempts, one hypothesis, and a termination rule written down before either attempt begins. A spike isn't a miniature project; it's a bet with a deadline, and a bet that can't be lost isn't a bet at all.

Write the Hypothesis Before the Clock Starts

A spike begins with one sentence that predicts a specific, falsifiable outcome. Vague goals like "evaluate this assistant on our codebase" produce hours of pleasant activity and zero conclusions, because nothing in them can be refuted. The card should be short enough to read aloud before anyone types a prompt:

Hypothesis: the failing test retries_spec::replays_side_effect_once fails because
            the retry wrapper re-enqueues before the first attempt is acknowledged.
Prediction: a patch that acknowledges before re-enqueue makes the test pass twice
            on HEAD <sha>, touching the retry wrapper and nothing else.
Kill criteria: baseline test already green, two runs disagree on outcome, the two
            diffs share no file, or 90 minutes elapse.
Out of scope: production config, performance, security review.

Three properties make it useful. The base revision is pinned, so a moving branch can't contaminate the comparison. The prediction names an observable outcome rather than an intention, so "feels better" can never become a conclusion. The kill criteria can be checked by a script, which matters because the person running the spike is exactly the person most motivated to keep it alive.

Spike comes from Extreme Programming: a short, disposable experiment run before formal development to answer a technical question or reduce some area of uncertainty. It deliberately doesn't aim for mergeable code quality, only for the single piece of information "can this work?" Its opposite is the prototype—written to evolve into a product, whereas a spike is written to be thrown away. The practical test for telling them apart: if this exploration fails, will you delete the code it left behind? If yes, it was a spike.

Another key concept is falsifiability, drawn from Popper's definition of a scientific proposition: a proposition carries information only if it could in principle be overturned by evidence. In an engineering context, "how does this assistant perform on our codebase" can't be overturned by any single run, since a good or bad result can always be explained as "not bad" or "try a few more times." Conversely, "a patch on a given commit that only touches the retry wrapper will make this test go green twice in a row" explicitly specifies the scenario that would falsify it—precisely why it's useful.

A Single Passing Run Is Basically a Coincidence

A single attempt that turns a red test green only tells you that something worked once. It can't tell you whether the assistant found the root cause or merely sidestepped the assertion, and in the terminal those two futures look identical. Cheap sampling makes both futures visible: on the same revision, in two disposable worktrees, run the same unmodified prompt twice, then compare the results and diffs side by side.

Two runs can't prove correctness, and this protocol never claims they can. They mark how much confidence a 90-minute budget can honestly buy, and they turn a matter of taste into a pair of files on disk. When the two runs disagree, the disagreement is itself the finding: the prompt, the test, or the model is unstable, and none of the three should be merged.

The discipline that makes this work is refusing to coach the second run. If the first attempt fails and the operator rewrites the prompt, pastes in a stack trace, or points at a suspicious file, the two runs are no longer samples of the same process. The second run then demonstrates the operator's understanding—which was never the question at hand.

Git worktree is a mechanism Git has provided since 2.5: it allows multiple working directories to be mounted on the same repository, each checking out a different branch or commit, while sharing the same object store and refs. Compared with git clone into a temp directory, a worktree is created almost instantly, doesn't re-download history, and doesn't pollute the remote. Running git worktree add --detach <path> HEAD gives you a clean checkout at detached HEAD—it isn't bound to any branch name, so at the end of the spike you can simply discard it without worrying about accidentally pushing.

This is the technical foundation that lets two independent runs proceed in parallel. Without worktrees, operators tend to settle for less: repeatedly git stash-ing and switching back and forth in a single checkout, so the second run inevitably inherits leftover state from the first, and the two samples are no longer independent. Worktrees make "both runs start from exactly the same point, without interfering with each other" a mechanical guarantee rather than a mere discipline, and you can clean up afterward with git worktree remove --force plus git worktree prune.

A Harness That Shares One 90-Minute Clock

The script below creates worktrees, records the base revision, checks that the target test actually fails before the assistant touches anything, and saves each attempt's patch, diff stats, and test result. A single deadline is computed at the start and shared by both runs, so neither run gets a private 90 minutes.

#!/usr/bin/env bash
# spike.sh - two-run, time-boxed spike harness. Illustrative: adapt the paths and commands.
set -euo pipefail

TEST_CMD=${TEST_CMD:-"npm test -- --runInBand"}
ASSISTANT_CMD=${ASSISTANT_CMD:-"./scripts/ask-assistant.sh"}  # your client, your model
PROMPT=${PROMPT:-"spike/hypothesis.md"}
BUDGET=${BUDGET:-5400}              # 90 minutes of wall clock for the whole spike
RUNS=${RUNS:-2}

OUT="spike/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
deadline=$(( $(date +%s) + BUDGET ))
left() { echo $(( deadline - $(date +%s) )); }

for n in $(seq 1 "$RUNS"); do
  wt="$OUT/wt$n"
  git worktree add --detach "$wt" HEAD >/dev/null 2>&1
  git -C "$wt" rev-parse HEAD > "$OUT/run$n.base"

  if timeout "$(left)"s bash -c "cd '$wt' && $TEST_CMD" > "$OUT/run$n.before" 2>&1; then
    echo "run$n: baseline already green - hypothesis rejected" > "$OUT/verdict.txt"
    exit 2
  fi

  timeout "$(left)"s bash -c "cd '$wt' && $ASSISTANT_CMD < '$PROMPT'" > "$OUT/run$n.answer" 2>&1 || true

  git -C "$wt" diff      > "$OUT/run$n.patch"
  git -C "$wt" diff --stat > "$OUT/run$n.stat"
  if timeout "$(left)"s bash -c "cd '$wt' && $TEST_CMD" > "$OUT/run$n.after" 2>&1; then
    echo pass > "$OUT/run$n.result"
  else
    echo fail > "$OUT/run$n.result"
  fi
done

python3 spike/verdict.py "$OUT"

Two details are easy to get wrong. timeout measures wall clock, so a run stuck on a prompt can't borrow time from the next run. The worktree sits detached at HEAD, which means the spike never touches the branch a human is working on, and you clean up afterward with git worktree remove --force and git worktree prune.

Two Records, Four Possible Verdicts

The reduction step is deliberately blunt. It reads the two result files, the two base revisions, and the list of files from the diff stats, then emits exactly one verdict, with no weighting and no request for a summary.

# spike/verdict.py - reduce two spike runs to one verdict.
import json, pathlib, sys

out = pathlib.Path(sys.argv[1])
runs = {}
for result_file in sorted(out.glob("run[0-9]*.result")):
    name = result_file.name.split(".")[0]
    runs[name] = {
        "result": result_file.read_text().strip(),
        "base": (out / f"{name}.base").read_text().strip()[:12],
        "touched": sorted((out / f"{name}.stat").read_text().splitlines()),
    }

outcomes = [run["result"] for run in runs.values()]
bases = {run["base"] for run in runs.values()}
overlap = set.intersection(*(set(run["touched"]) for run in runs.values())) if runs else set()

if len(runs) < 2:
    verdict, why = "kill", "the spike did not finish both runs inside the budget"
elif len(bases) > 1:
    verdict, why = "kill", "runs started from different revisions"
elif outcomes == ["pass", "pass"] and overlap:
    verdict, why = "ship-to-review", "both runs passed and touched at least one identical file"
elif outcomes == ["pass", "pass"]:
    verdict, why = "kill", "two passes with disjoint diffs; the cause was not identified"
else:
    verdict, why = "kill", "runs disagree on outcome"

(out / "verdict.json").write_text(json.dumps({"verdict": verdict, "why": why, "runs": runs}, indent=2))
print(verdict, "-", why)

The verdict table looks like this:

Run 1	Run 2	Diffs share a file	Verdict
pass	pass	yes	ship to review
pass	pass	no	kill
pass	fail	either	kill
fail	fail	either	kill, and rewrite the hypothesis
any	unfinished	-	kill, with no partial credit

Harshness is the point. A spike that ends with a patch, two records, and a written reason to stop has produced real value, because the next person won't spend another afternoon on the same argument. "Ship to review" isn't "ship": the patch still goes through the normal review process, and reviewers will still raise questions the two-run protocol can't answer.

How Free Model Access Changes the Arithmetic

The protocol above doubles the number of attempts and test executions, which is precisely the kind of cost that pushes teams to sample once and call it evidence. The open source project MonkeyCode currently offers free model access and a free server option, which removes the per-attempt price argument from the decision and makes the second run routine rather than a favor.

Disclosure: this article was prepared as part of a promotion for the MonkeyCode product.

Two honest caveats. Free tiers (including any token quotas) are defined by the project and change over time, so read the current terms on the project's page rather than planning around numbers quoted in a blog post. The assistant invocation in the harness is deliberately left as a placeholder: the interface for talking to a hosted model is a project detail, so check the repository for its current client shape instead of assuming a command-line tool that may not exist.

Where the free server option genuinely participates is the clock. If the harness runs on a machine that isn't the developer's laptop, a slow test suite no longer competes with the editor, and the 90 minutes go where the work actually happens. Long runs can be detached and tracked with logs, but the deadline must stay attached to the process.

# Illustrative: detach the spike so the 90-minute clock outlives the terminal session.
cd ~/repo && PROMPT=spike/hypothesis.md nohup ./spike.sh > spike.log 2>&1 &
tail -f spike.log

Limits, and Who Should Skip This Entirely

The protocol measures agreement between two samples, nothing more. It can't distinguish a flaky test from an unstable model, so a test suite that already fails intermittently will produce a kill for the wrong reason; stabilize the flaky test first, or the spike is just noise. It knows nothing about performance, security, or behavior under production load, and it can never be a reason to skip review.

Some teams simply shouldn't use it. Someone who needs a fix today should fix it today, because a spike is the wrong tool for an outage. Teams in regulated environments that require auditable tools and data handling agreements should confirm those constraints before sending code to any hosted assistant. And anyone evaluating models rather than bugs should pick a different design, because running twice on one test is too small a sample for that question.

The useful version of this workflow ends with a small evidence directory: two patches, two base revisions, two test logs, and one verdict with a reason. If a spike whose marginal cost is zero sounds like the interesting part, free model access and a free server option are the cheapest way to find out whether your hypothesis survives its first honest disagreement.

One-line Summary

Turn team arguments about AI patches into decidable evidence using two independent runs, a falsifiable hypothesis, and a hard 90-minute deadline.

Paragraph Summary

The article addresses the dilemma of teams arguing endlessly while AI coding assistants fix bugs, and proposes a time-boxed experiment protocol called a spike. Its core is converting vague disputes into a falsifiable hypothesis, an explicit prediction, and termination criteria locked in before starting, then running twice independently with the identical prompt in two isolated Git worktrees within a single shared 90-minute clock. Only when both runs pass and the patches touch the same file is the verdict ship-to-review; every other combination—disagreeing outcomes, disjoint diffs, drifting base revisions—is an automatic kill. The author provides a bash script and a Python verdict reducer to demonstrate the full workflow, and stresses that "ship to review" is not the same as merging. Finally, the article notes that the protocol only measures agreement between two samples, can't distinguish flaky tests from unstable models, and doesn't apply to emergency fixes, regulated environments, or model evaluation.