Catching Weakened Tests with Three Git Worktrees

Title

Catching Weakened Tests with Three Git Worktrees

Meta Description

Use three Git worktrees to simulate baseline, patched, and mutant states, mechanically verifying whether tests in a patch actually observe the fix, with complete shell scripts, decision tables, and failure mode analysis.

Article Content

The patch passed CI, the diff reads clean, and review took ninety seconds. Six weeks later, a pagination regression slips through the exact code path guarded by the assertion that was loosened at the time. This isn't a specific incident—it's a reproducible scenario.

Why a Green Test Suite Proves Little

A passing test suite answers only one narrow question: the existing assertions didn't fail. It doesn't say whether the assertions are strict enough, nor whether the new tests would also pass on unfixed code. Reviewers typically fill this gap with attention—and attention is exactly the resource that runs out first on a Friday afternoon.

The original article proposes using three cheap signals to restore the missing information: whether the new test fails on baseline; whether the suite fails when the fix is removed but the tests are kept; and how many assertions the patch deletes or loosens.

Three States Every Patch Should Go Through

Each state is a detached Git worktree—no branch switching, no stash risk:

1. Baseline state: upstream tip plus the contributor's test changes. New tests must fail here, and the failure reason must be the expected one.
2. Patched state: apply the contributor's full diff. The suite must pass.
3. Mutant state: on the patched tree, revert src/ to baseline while keeping tests as-is. The suite must fail—otherwise the tests never observed the fix.

The third state is the one most review workflows skip, and the only one that can identify tests that "describe the fix" rather than "constrain the fix."

Git worktree is a command introduced in Git 2.5 that allows the same repository to check out multiple working trees into different directories simultaneously. Each worktree independently points to a different branch or commit (usually in a detached HEAD state), sharing the same .git object database. Compared to git stash or frequent branch switching, the core advantage of worktrees is: you can inspect code at multiple commits in parallel without staging uncommitted changes or interrupting your current work state. In the three-state check described here, the three worktrees correspond to baseline, patched, and mutant states, isolated from each other so test runs don't interfere. Destroying a worktree only requires git worktree remove or simply deleting the directory followed by git worktree prune—ideal for "use and discard" review scenarios.

Mutation testing is a method for evaluating test suite effectiveness: artificially inject defects (mutations) into code, then run tests to check whether they detect these defects. If tests still pass on the mutant, the tests don't truly constrain code behavior. The "mutant state" in this article is a lightweight special case of mutation testing: instead of injecting random mutations, it directly reverts the entire fix (src/ back to baseline) while keeping tests unchanged. This constructs a "known-buggy" mutant—if the test suite can't detect it, the new tests didn't observe the fix. Compared to traditional mutation testing tools (like PIT, mutmut), this approach has extremely low computational cost (running the suite only once), but it can only verify "whether the fix is constrained by tests," not comprehensively assess overall test suite strength.

Building a Ten-Minute Reproducible Fixture

The fixture is deliberately small: one module, one test file, one bug. The bug is a missing ceiling division—any total smaller than the page size returns zero pages.

mkdir tamper-demo && cd tamper-demo
git init -q
mkdir -p src tests bin

The baseline src/paging.py has just one line: total // size. The baseline tests include three cases: exact division, large total, and zero total. Then create a patch branch that fixes the division, adds a fourth test, and changes two existing assertions to looser >= comparisons. The new test uses pages(10, 5) == 2—which also passes on buggy code, so it catches nothing.

The Skeleton of the Three-State Check Script

The script assumes pytest is installed and that src/ is the only directory carrying the fix. The core logic is three worktrees plus three verdicts:

git worktree add -q --detach "$WORK/base"    "$BASE_REF"
git worktree add -q --detach "$WORK/patched" "$PATCH_REF"
git worktree add -q --detach "$WORK/mutant"  "$PATCH_REF"
git -C "$WORK/mutant" checkout -q "$BASE_REF" -- src/

If the baseline state is all green, print "WARN base is green; the failing reproduction is missing"; if the patched state has failures, mark FAIL; if the mutant state has no failures, the tests observed nothing—mark FAIL. Finally, use git diff -U0 to count added and removed assert lines in test files; when the removal count is greater than zero, prompt to read the test diff first.

git checkout <commit> -- <path> is an easily confused form of a Git command: it extracts path contents from the specified commit, overwriting the corresponding files in the working tree and staging area, but without moving HEAD. In the script here, git -C "$WORK/mutant" checkout -q "$BASE_REF" -- src/ does the following: in the mutant worktree, restore the src/ directory to the baseline version while keeping test files and other files unchanged. This precisely constructs the "fix reverted, tests kept" state. Note that this command directly modifies working tree files; if you want to restore the patched state afterward, you need to checkout back to the patch commit again or use git checkout -- .. This usage is lighter than git revert because it creates no new commit and doesn't affect branch history.

Reading the Output Like a Reviewer

Running the script on the fixture, the report looks roughly like this:

== state 1: base (main) ==
1 failed, 3 passed in 0.11s
== state 2: patched (patch) ==
4 passed in 0.09s
== state 3: mutant (fix reverted, tests kept) ==
4 passed in 0.09s
FAIL the suite passes without the fix; the tests observe nothing
== assertion churn ==
assertions added=1 removed=2
REVIEW the test diff first: 2 assertion line(s) removed

Two independent signals point to the same weakness: the mutant run proves the new test can't distinguish fix from breakage, and the churn count shows two existing guarantees were weakened. Either alone isn't conclusive, but together they're enough to support "request changes" rather than "approve."

The failure message itself matters as much as the failure. If the mutant run fails on an import error, it proves nothing about assertions—you must see AssertionError or a specific test name in the log for it to count.

What Free Models Can and Can't Help With

The original article draws the division of labor clearly. Models are suited for textual, low-risk parts: rewriting a failed mutant log into a two-sentence PR comment, drafting missing tests from the diff, naming the observable behavior the patch changes, listing existing assertions that a refactor reasonably touches. The judgments reserved for maintainers are: whether a weakened assertion is justified, whether the suite is flaky or genuinely wrong, and whether to merge.

A prompt skeleton that keeps models within bounds: given the unified diff, baseline log, patched log, and mutant log, don't ask for merge decisions; list the observable behaviors the diff changes; for each new test, state whether it would fail without the diff; propose one additional test that would fail in the mutant state; mark anything undeterminable from logs as unknown.

The original article discloses its association with MonkeyCode here: the article is part of its product promotion, and the capacity and terms of the free model access and free server options are provided by the operator and may change. Disposable hosts are genuinely convenient here—three worktrees plus virtualenvs plus test caches accumulate quickly, and destroying them after running makes "starting from a clean state" the default rather than a burden; copying back only the report logs also keeps review artifacts small and auditable.

Patch Triage Decision Table

Signal	Merge	Ask Author	Block
New test fails on baseline, passes on patch, fails on mutant	Yes	—	—
Mutant state passes	—	—	Yes
Assertion lines deleted	—	Yes, require written justification	—
Test file exceeds 30% of diff	—	Yes	—
Baseline already all green	—	Yes, reproduction missing	—

The value of this table is converting script output into a repeatable first response—more important than any single verdict.

Limitations and Failure Modes

The mutant run is only a heuristic, and it degrades in predictable ways. Reverting src/ may break imports, and mutant failures may stem from something other than the assertion under test; suites asserting on mocks or log strings often stay green even when behavior changes; flaky tests make the third state noisy, requiring retry strategies to trust the signal; large monorepos need affected-test selection, which narrows mutant run coverage to the selected subset; patches that legitimately rename or refactor tests will trigger the churn counter and need human reading.

Flaky tests are tests that pass sometimes and fail other times when running the same code in the same environment multiple times. Common causes include: dependence on time or timezone, dependence on random numbers, dependence on network or external services, shared state between tests not cleaned up, concurrency races, etc. In mutant state checks, flaky tests create noise: even if the fix is genuinely constrained by tests, the mutant run might be misjudged as "tests effective" due to a flaky failure; conversely, if a flaky test happens to pass, it might mask the fact that tests have no constraining power. The original article suggests "requiring retry strategies to trust the signal"—typically running the mutant state multiple times (e.g., 3 times) and only accepting a valid signal when all fail. A more fundamental solution is to first isolate and fix flaky tests before incorporating them into the three-state process, otherwise the entire check's credibility will continuously decay.

Who Should Skip This Process

Repositories without a usable test command shouldn't start here—the script only amplifies an already-existing suite. Teams that treat tests as documentation rather than constraints will find the third state always red and will turn it off within a week. Maintainers who review one small patch a month will find carefully reading the test diff more valuable than maintaining three worktrees and a shell script. The benefits of this process appear in repositories where patch volume exceeds reviewer attention and where weakened assertions become more costly the later they're discovered. Three worktrees, one script, one log file—enough to make this check routine.

One-line Summary

Three-state patch checking with three Git worktrees identifies weakened tests and ineffective assertions.

Paragraph Summary

The article points out that a green test suite only proves existing assertions didn't fail—it can't guarantee assertions are strict enough or that new tests can distinguish fix from breakage. It proposes a three-state check: baseline state requires new tests to fail, patched state requires passing, mutant state (revert src, keep tests) requires failing. Through a ten-minute reproducible fixture and shell script, it demonstrates how to set up three detached worktrees with automatic verdicts, while counting deleted assert lines in the test diff as a second signal. The article also discusses free models'辅助 role in text rewriting and test drafting, the various failure modes of mutant runs as heuristics, and provides a patch triage decision table and applicability boundaries.