AI Refactor Claims Nothing Changed? Verify with Dual-Branch Differential Testing

AI coding agents often confidently report "no behavior changes, all tests passing" after completing a refactor. But passing tests only prove they didn't break the parts being checked—hidden behavioral differences may be quietly slipping into production. This article introduces a ~50-line bash differential testing script: use git worktree to run pre- and post-refactor code side by side, feed them identical real inputs, and compare outputs line by line, making the agent's refactor promises verifiable.

Tests Passing Doesn't Mean Behavior Unchanged

A typical example: the agent moved invoice total calculation from InvoiceController into an InvoiceTotals use case, claiming "no behavior changes, all tests passing." But the old code rounded each line amount to cents before summing, while the new code added raw amounts first then rounded at the end. For an invoice with three line items of 0.30 net each at 19% tax, the old algorithm rounds each line's 0.357 to 0.36, totaling 1.08; the new algorithm sums to 1.071 and rounds to 1.07. If all test fixtures have single-line invoices, both versions produce the same result, the test suite stays green, and the bill is a cent short.

GitHub's Scientist library states it plainly in its README: "Tests can guide refactoring, but what you really need is to compare current behavior with refactored behavior under load." Differential testing is the local lightweight version of this idea.

Step 1: Isolate Baseline Code with git worktree

git worktree is a second working directory linked to the same repository. You need one branch for the agent, another for the commit where the agent started. The key is comparing against the starting commit, not today's main—if main moved while the agent worked, comparing against it would mix in others' changes. Use git merge-base to find the branch fork point:

cd ~/code/app
base=$(git merge-base main hermes/extract-invoice-totals)
git worktree add --detach ../app-base "$base"
cp .env ../app-base/.env
(cd ../app-base && composer install -q)

Passing a commit hash instead of a branch name bypasses git's restriction that "a branch can only be checked out in one worktree." The new worktree contains only tracked files—ignored files like .env and vendor/ need to be copied or installed.

git merge-base is the git command for finding the best common ancestor between two commits. In branch scenarios, it returns the commit where the two history lines diverged, not the current tip of main. Suppose the agent branched from commit C1 on main, and during that time main advanced to C3. git merge-base main hermes/... still returns C1. This way the baseline worktree precisely corresponds to the code state when the agent started working, avoiding misattributing others' merged changes as behavior changes introduced by the agent. If there are multiple fork points (history has multiple merges), use --fork-point or specify multiple candidate branches to refine.

Step 2: Record Production-Like Inputs

The cases file has one shell command per line, each run once in both worktrees. Comment lines starting with # are skipped. Cases should cover mundane scenarios, multi-line scenarios, missing IDs, and malformed inputs:

# tests/diff/cases.txt
php artisan invoice:show 1001 --json
php artisan invoice:show 1002 --json
php artisan invoice:show 1003 --json --currency=EUR
php artisan invoice:show 999999 --json
curl -s -w "\n%{response_code}\n" "$APP_URL/api/invoices/1001"
curl -s -w "\n%{response_code}\n" "$APP_URL/api/invoices/abc"

HTTP inputs don't need to be invented from scratch. nginx's default combined log format puts method and path in fields 6 and 7, so real GET paths can be extracted directly from access logs:

awk '$6 == "\"GET" { print $7 }' access.log \
  | sort -u | head -200 \
  | sed 's|.*|curl -s -w "\\n%{response_code}\\n" "$APP_URL&"|' \
  >> tests/diff/cases.txt

Manually review the results before committing: query strings may contain tokens and personal data, and $ in paths will be expanded by the shell. Unless both sides have independent database copies, keep only GETs—replayed POSTs will change data rows that subsequent cases read.

Step 3: Core Logic of the Differential Script

In the script, stdout and stderr are written to the same file, and non-zero exit codes are appended as exit=N, so newly appearing warnings or exit code changes count as differences. Two safeguards prevent "passing without running anything": if the cases file contains HTTP lines but the URL isn't set, error out immediately; if curl exits non-zero (e.g., connection failure), the case fails outright, because two identical errors look like a match to diff.

When output parses as JSON, jq -S sorts keys—a refactor may change array construction order without changing what JSON clients see. Plain text, empty files, or exit=N lines go through cat for direct comparison. Then sed masks ISO timestamps and UUIDs, and anything else that changes every run goes into JQ_FILTER:

export JQ_FILTER='walk(if type == "object"
  then del(.request_id, .generated_at) else . end)'

HTTP cases need a server running in each worktree (e.g., ports 8001 and 8002), passing the addresses as BASE_URL and CAND_URL respectively.

Before trusting the diff, let the old version compare against itself. In a self-comparison run pointing at the same server, every difference is noise—eliminate it with normalization rules until clean. The Diffy tool does the same thing with three instances: candidate, primary, and secondary (last known good code). When primary and secondary disagree, "Diffy treats that signal as noise."

jq's walk function recursively traverses every level of JSON, applying the given expression to each value, commonly used for global cleanup. In the example, walk(if type == "object" then del(.request_id, .generated_at) else . end) means: when encountering an object, delete the specified volatile fields; return other types as-is. Combined with jq -S (i.e., --sort-keys) to sort by key name first, it eliminates pure formatting differences caused by different object key insertion order. Note that walk rebuilds the entire structure by default; if you need to preserve original key order, use del with path expressions instead. Also, when using -S and walk together, sorting happens at output stage and doesn't affect deletion logic.

Step 4: Make the Diff a Task Completion Condition

This is where Hermes IDE comes in (the author is involved in its development—weigh this context accordingly). Hermes starts each task in a separate hermes/<name> branch and worktree. The task itself is the agent's first prompt, and the agent can be Claude Code, Codex, or any other CLI agent.

Hermes reads the .hermes/worktree.toml recipe in the repository, where the done_when list holds the task's default checks:

# .hermes/worktree.toml
setup = ["composer install -q"]
copy = [".env"]
done_when = [
  "composer test",
  """BASE_URL=http://127.0.0.1:8001 \
  CAND_URL=http://127.0.0.1:8002 \
  ~/code/app-base/scripts/difftest.sh ~/code/app-base .""",
]

Triple quotes are TOML's multi-line string form, and trailing backslashes strip newlines and indentation, making the second check a single command. When the script is run without a third argument, it reads cases from the baseline folder—so both the script and cases come from the baseline worktree outside the agent's worktree. Otherwise, an agent told to "make this check pass" might just edit cases.txt until the diff goes quiet.

Delta checks should only be used on tasks that promise no behavior changes. Feature task diffs should be non-empty by design—a "fail on any difference" check would block the changes you asked for. For those tasks, run the script manually and use the diff as a review aid.

Where This Approach Breaks Down

The diff only covers the inputs you recorded. If the cases file has no multi-line invoices, the rounding change from the opening example slips through. Spend effort on the cases file—add a case every time a production bug appears.

Paths with side effects need isolation. Emails, queued jobs, payments, shared database writes will all run twice, once on each side. Point both worktrees at fakes or separate copies, or leave these paths to the test suite.

Non-determinism you can't mask will keep the check red. Unordered lists from the database can be sorted in JQ_FILTER, random sampling needs a fixed seed, and results depending on wall-clock time should have that case removed. Timestamp regexes are also crude—they mask every datetime in the output, including the one a refactor might have broken. If dates matter to the changed code, mask by field name in JQ_FILTER instead.

Diffy is Netflix's open-source differential testing service, built around three roles: candidate, primary, and secondary (the last known good version). It sends the same request to all three instances simultaneously. Only when the candidate disagrees with the primary while the primary agrees with the secondary does it judge that the candidate introduced a regression; if primary and secondary themselves differ, that difference is marked as noise and discarded. This design automatically filters out non-deterministic output that already exists in production, making it more robust than simple "old vs. new" comparison. The local script has no third instance, so it achieves a similar effect with "old version self-comparison": run the same code twice first, normalize all non-deterministic differences to zero, then trust the old-vs-new comparison.

Key Takeaways

When an agent says "no behavior changes," that's a claim to verify, not a conclusion. Passing tests prove "didn't break what was checked"; differential testing adds "didn't change anything that was recorded." Write the diff check into done_when, and refactor tasks shift from "I trust the agent" to "the evidence says it's mergeable."

One-line Summary

Use git worktree and a ~50-line bash differential script to make the agent's "no behavior changes" refactor promise verifiable.

Paragraph Summary

The article points out that AI coding agents often claim "no behavior changes, tests passing" after refactoring, but passing tests only prove nothing checked was broken—hidden behavioral differences (like changes in rounding order) can escape an all-green test suite. The author proposes a ~50-line bash differential testing script: use git merge-base to locate the branch fork point, git worktree --detach to create a baseline code directory, record production-like inputs (CLI and HTTP requests), run them side by side in both worktrees, compare output line by line, and use jq sorting and sed masking of timestamps and other non-deterministic content. Before trusting the diff, the old version must be self-compared to eliminate noise. This method only applies to refactor tasks that promise no behavior changes, side-effect paths need isolation, and coverage is limited by the recorded inputs.