Before You Let an AI Coding Agent Fix a Flaky Test, Make It Fail Reliably

You tell your AI coding agent "fix that checkout test that fails sometimes," it runs the test twice, both are green, and it reports the task complete. But two passes is exactly what the broken version looks like most of the time—the agent has proven nothing. This article is about how, before handing the task to an agent, to make a flaky test fail reliably on demand, write down a reproducible command and seed, and then hand the agent a check that must pass hundreds of times in a row.

The problem: "looks done" is the agent's only signal

Claude Code's best-practices page states plainly: "Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available." Flaky tests are the worst case for "looks done"—they pass most of the time, and the agent is easily misled.

A reproducible failure is not an occasionally-red CI link; it's one command plus one seed that fails reliably within a few minutes. To get there, you first need to figure out which kind of flake you're dealing with.

Two kinds of flaky tests, two ways to find them

The first kind fails under any order: data races, too-tight timeouts, clock issues, shared state within a single test. You expose it through volume—run the same test hundreds of times.

The second kind depends on what ran before: polluted globals, database rows left by other tests, caches. You expose it through order—run with many different orders, each backed by a replayable seed. The article's illustration captures this well: in a long row of black checkmarks, a single cracked orange block hides under a magnifying glass.

Go: -count repeats the order, -shuffle decides the order

go test supports both. -count n runs each test n times, -shuffle on randomizes test order and reports the seed so you can reproduce it, and -race enables the race detector. Start with a volume test on a single test:

go test ./internal/cart -run '^TestCheckoutTotals$' \
  -count=500 -race -failfast -timeout=30m

-failfast stops on the first failure, keeping output short. -timeout matters too: the default is 10 minutes, and the race detector docs say typical programs "may use 5-10x more memory and 2-20x more execution time." 500 runs with the race detector exceeding 10 minutes makes the test binary panic, which looks like yet another failure unrelated to the bug.

If this test never fails on its own, it may depend on other tests, and then you need order. There's a detail in the Go source that changes how you write the command: in testing.go, M.Run shuffles only once for top-level tests, prints the seed, and only then enters the -count loop. So -count=500 -shuffle=on is one random order run 500 times, not 500 orders.

To get multiple orders, run multiple test binaries, each with your own chosen seed:

log=$(mktemp)
for i in $(seq 1 50); do
  seed=$RANDOM
  if ! go test ./internal/cart -count=10 \
      -shuffle="$seed" -race >"$log" 2>&1; then
    echo "FAILED: -shuffle=$seed"
    tail -n 40 "$log"
    break
  fi
done

On failure, go test ./internal/cart -count=10 -shuffle=<seed> -race replays the same order. There's another easy trap: Go caches passing results when you specify packages, so if the agent runs go test ./internal/cart twice with no changes in between, the second run may get a cached pass rather than a real run. The docs call -count=1 "the idiomatic way to explicitly disable the test cache." Every command you hand your agent should include -count.

The race detector (-race): Go's built-in dynamic analysis tool inserts synchronization-event detection code at compile time and, at runtime, records all memory accesses and goroutine synchronization operations, reporting data races and exiting with a non-zero status when it finds unsynchronized concurrent reads/writes. The official docs note that enabling it typically increases memory use 5–10x and execution time 2–20x, so -timeout must be raised accordingly—otherwise the test binary panics on timeout, producing a failure unrelated to the target bug. The race detector only catches races on paths actually executed; the more repetitions, the higher the chance of hitting the window, which is exactly why you lean on -count to amplify execution volume.

pytest: pytest-repeat handles volume, pytest-randomly handles order

Python needs two plugins. pytest-repeat provides --count, and its README notes you can pair it with -x to "make the test runner stop on the first failure." pytest-randomly shuffles test order, prints the seed at the start (its example is Using --randomly-seed=1553614239), and resets the global Python random seed "before the setup, run and teardown of each test." So one seed replays both the order and any data drawn from random.

python -m pip install pytest-repeat pytest-randomly
pytest "tests/test_cart.py::test_checkout_totals" \
  --count=500 -x -q

Same trap as Go: a single pytest run has only one seed, so all 500 copies run under the same shuffled order. To get multiple orders, drop the following into the loop from the Go section, passing the seed yourself:

pytest tests/ -x -q --randomly-seed="$seed"

--randomly-seed=last reruns the previous seed, provided pytest's cache provider plugin is enabled.

Then disable the plugin and run each test 50 times in file order: pytest tests/ -x -q -p no:randomly --count=50. Don't add -p no:randomly to the loop, because with the plugin off pytest no longer recognizes --randomly-seed and every run exits with a usage error. If it fails when shuffled but never in file order, the test depends on residue from other tests; if it fails both ways, look at time, threads, network, or the database.

pytest's plugin mechanism and the cache provider: pytest discovers third-party plugins via setuptools entry points, extending command-line options and hooks without code changes. pytest-randomly uses this mechanism to register the --randomly-seed option and resets the global random seed before and after each test's setup/run/teardown, ensuring the data drawn from random is reproducible under the same --randomly-seed. --randomly-seed=last relies on pytest's built-in cache provider plugin (cacheprovider) to store the last seed; if you disable caching with -p no:cacheprovider, the option breaks. Likewise, -p no:randomly fully unloads the plugin, so pytest no longer recognizes its command-line options—exactly why you can't add that flag inside the loop.

Vitest and Jest: choose the seed yourself

Vitest 5 can do both from the command line. --repeats <number> "repeats each test the specified number of times regardless of the result," and --sequence.shuffle with --sequence.seed gives a seeded order. The sequence config docs say the seed defaults to Date.now() but don't say Vitest prints it. So pass it yourself, and you'll always know which one failed:

npx vitest run src/cart --repeats=50 --bail=1
npx vitest run --sequence.shuffle \
  --sequence.seed="$seed" --bail=1

Jest 30 has --randomize, which shuffles tests within files and shows the seed, and only works with the default jest-circus runner. --seed=<num> sets the seed, and Jest also uses it to shuffle test-suite run order; the value must be between -2147483648 and 2147483647. Bash's $RANDOM ranges from 0 to 32767, so it fits. For Jest, the loop handles repetition:

npx jest src/cart --randomize --seed="$seed" --bail

Seeds and pseudo-random number generators: shuffle order is produced by a pseudo-random number generator from a seed, and the same seed always yields the same permutation—that's the basis of "replayable." Vitest's --sequence.seed defaults to Date.now() but isn't echoed, and Jest's --seed is limited to 32-bit signed integers, erroring if exceeded. Bash's $RANDOM returns 0–32767, safely inside Jest's range, but with low entropy—large loops can produce duplicate seeds, and duplicate seeds mean duplicate orders, diluting the benefits of multi-order coverage. For order-dependent flakes, use $RANDOM$RANDOM or draw from /dev/urandom to widen the seed space.

The check script and the task brief

Put the loop in a script that exits 0 only if all runs pass. You run it, the agent runs it, and nobody has to judge "looks fine."

#!/usr/bin/env bash

**The Stop hook and the agent's completion decision**: Claude Code's Stop hook is a user-defined script triggered when the agent is about to end a turn; a non-zero exit blocks the turn from ending and feeds the output back to the agent, forcing it to keep working. This gives "done" an external enforcement point: wire `scripts/flaky-check.sh` to the Stop hook, and the agent can't call it a day on "tests look green"—it must make the script exit 0. Note that the hook script itself should fail fast and limit output length, or the agent may burn a lot of context on repeated retries; and the hook doesn't replace human review—you still need `git diff` to confirm the agent didn't touch assertions or the check script itself.
# scripts/flaky-check.sh
# Exit 0 only if RUNS seeded runs all pass.
set -u
RUNS=${RUNS:-50}

run_once() {
  go test ./internal/cart -count=10 -shuffle="$1" -race
}

log=$(mktemp)
for i in $(seq 1 "$RUNS"); do
  seed=$RANDOM
  if ! run_once "$seed" >"$log" 2>&1; then
    echo "run $i/$RUNS FAILED, seed $seed"
    tail -n 40 "$log"
    exit 1
  fi
done
echo "$RUNS/$RUNS runs passed"

50 runs of -count=10 is 500 executions of each test in the package across 50 orders. For pytest, swap the body of run_once for pytest tests/ -x -q --count=10 --randomly-seed="$1"; for Vitest or Jest, use the commands above with "$1" as the seed.

How many runs you need depends on the failure rate, so measure with the loop first. If it fails once in 50 executions and the "fix" changed nothing, then the probability that 500 executions in the script (50 runs × -count=10) all pass is 0.98^500, about one in twenty-four thousand. If the failure rate is one in a thousand, a still-broken test has a 61% chance (0.999^500) of passing those 500 executions. Rare flakes need bigger numbers. Order-dependent flakes don't have a stable per-execution failure rate, because they only fail in certain orders, so for those, raise RUNS to try more orders.

Then comes the task brief. The 8812 below stands for the seed your loop printed:

Fix the flaky test TestCheckoutTotals in ./internal/cart.

Reproduce first. This fails within a few minutes:
  go test ./internal/cart -count=10 -shuffle=8812 -race
Run it before you change anything and confirm it fails.

Failure:
  <paste the failing assertion and any race report>

Rules:
- Find the cause. No retries, sleeps, t.Skip or
  longer timeouts.
- Don't change the test's assertions or
  scripts/flaky-check.sh.

Done when both pass:
  go test ./internal/cart -count=10 -shuffle=8812 -race
  scripts/flaky-check.sh

Reproduce first is Claude Code's own advice for bugs: "write a failing test that reproduces the problem, then fix it." If you want to enforce that gate inside Claude Code, the same docs describe a Stop hook that "runs your check as a script and blocks the turn from ending until it passes."

Running the loop alongside the agent in Hermes

(I develop Hermes IDE, so weigh this section accordingly.) Everything above works in any terminal. Hermes puts the agent, your reproduction loop, and the verdict in one window.

Press ⌘N (Ctrl+Shift+N on Windows and Linux) and paste the task brief in. The agent—Claude Code or Codex, say—starts in a real terminal with your own login, on its own hermes/<name> branch and worktree, with the brief as its first prompt.

Then split the pane right or down and open a plain shell next to the agent. git worktree list shows the task folder; cd into it and run the seeded reproduction manually once before the agent's first edit and once after its last. One caveat: your loop and the agent's own test runs compete for CPU, which changes timing. Races may show up more or less often when both are busy, so run the final check while the agent is idle.

The script is always the definition of "done," so run it in that pane before landing. Then open Review Desk (⌘G, or Ctrl+Shift+G) and read the changes turn by turn, checking the rules were followed: no new sleeps, no retries, no loosened assertions. For the check script, a plain git one-liner answers it:

git diff main...hermes/<name> -- scripts/

Empty output means the agent didn't touch your check.

Why this is worth doing

Flaky tests are a CI problem first and an agent problem second. Once the pipeline goes randomly red, people learn to click rerun, and real failures hide in the noise. Nailing the reproduction down to a command, a seed, and a script not only gives the agent a verifiable definition of "done"—it frees the whole team from guessing "looks fine."