Auditing AI Patch Testing: Don't Be Fooled by "42 passed"

When an AI agent's patch report says "42 passed," it only proves those 42 tests ran and passed — not that the entire test suite passed. When the repository actually collects 900 tests and the patch only executed 42, the remaining 858 tests are not evidence — they are just assumptions that haven't been priced yet.

This kind of scope claim needs to become auditable. The verification approach is to build a small evidence bundle paired with four gates: a selection/execution ledger, a reachability check from changed lines to owning tests, a closure check on fixture read sets, and a hash-frozen quarantine ledger, ensuring the patch cannot quietly expand the set of tests it is allowed to skip.

Three Kinds of Counts: Patches Usually Only Cite the Last One

A single test run produces at least three distinct numbers, and patches usually only report the last one.

Count	Stage produced	What it proves
collected	Collection stage	Tests still exist in the code tree
selected	-k, --lf, impacted-test tools	The scope the patch claims to cover
executed (call stage)	Run report	Test ran to a verdict
skipped / xfail	Run report	Before classification, proves nothing

The core gate is an inclusion check: executed ⊆ selected ⊆ collected. If a test was executed but never selected, the test framework is lying, and no verdict from that run can be trusted. If a changed line has no corresponding owning test in executed, the patch has unreachable changes that require a written justification.

pytest's collect / call stages: pytest's test run lifecycle consists of several hook stages, among which pytest_collection_modifyitems belongs to the collect stage — at this point test items have been discovered and organized into an items list, but not yet executed; while report.when == "call" in pytest_runtest_logreport represents the stage where the test function body is actually invoked and a verdict is produced. Between the two there are also setup and teardown stages. The reason for distinguishing these is that selection mechanisms like -k, --lf (last-failed), and --deselect all operate between collected and selected, while skips may occur at collection time (e.g., pytest.mark.skip or module-level skip) or at setup time. Without recording collected, selected, and executed in separate files, you cannot determine whether a test was "never collected" or "selected but conditionally skipped," and thus cannot audit the gap between "the scope the patch claims to cover" and the actual execution scope.

Step One: Record Selection and Execution Separately

The collection stage and the call stage are two different events with different payloads and should be written to different files.

# conftest.py — evidence recorder (reference implementation, adapt paths as needed)
import json, os, pathlib

ART = pathlib.Path(os.environ.get("EVIDENCE_DIR", ".evidence"))
ART.mkdir(parents=True, exist_ok=True)
COLLECTED = ART / "collected.jsonl"
EXECUTED  = ART / "executed.jsonl"

def pytest_collection_modifyitems(config, items):
    with COLLECTED.open("w") as fh:
        for item in items:
            fh.write(json.dumps({
                "nodeid": item.nodeid,
                "path": str(item.path),
                "markers": [m.name for m in item.iter_markers()],
            }) + "\n")

def pytest_runtest_logreport(report):
    if report.when != "call":
        return
    with EXECUTED.open("a") as fh:
        fh.write(json.dumps({
            "nodeid": report.nodeid,
            "outcome": report.outcome,
            "skipped": report.skipped,
            "duration": report.duration,
        }) + "\n")

After running the patch's own command, first check the inclusion relationship before reading the pass count. Skipped tests should be counted in executed, but must be counted separately. A patch converting failing tests into skipped tests only changes the verdict vocabulary, not the behavior.

Step Two: Map Changed Lines to Owning Tests

The patch's diff is the input; the per-test coverage contexts are the index. Record context for each test node, export JSON, and take the intersection.

coverage run -m pytest -q
coverage json -o .evidence/coverage.json

**Per-test coverage contexts**: coverage.py by default only records whether a line was executed, whereas `--cov-context=test` or `dynamic_context = test_function` attaches "in which test context was this line hit" information to each line, presented in the `files.<path>.contexts` field of the `coverage json` output, shaped like `{"12": ["tests/test_x.py::test_a|run"]}`. This is the key index linking "changed lines" to "owning tests." Note that the `contexts` key only exists when a file was actually executed by some test; files not touched by any test will have no contexts entry. Also, different coverage.py versions differ in context string format (whether `|run` is present, whether setup/teardown is included), so before computing diff line ownership you should first verify the actual JSON shape to avoid mistaking version differences for unreachable changes.
# reachability.py — reference implementation; verify JSON schema first
import json, re, subprocess, sys

diff = subprocess.run(["git", "diff", "-U0", "origin/main...HEAD"],
                      capture_output=True, text=True, check=True).stdout
HUNK = re.compile(r"^\+\+\+ b/(?P<path>.+)$"
                  r"|^@@ -\d+(?:,\d+)? \+(?P<start>\d+)(?:,(?P<count>\d+))? @@")

changed, path = {}, None
for line in diff.splitlines():
    m = HUNK.match(line)
    if not m:
        continue
    if m.group("path"):
        path = m.group("path")
        changed.setdefault(path, set())
    elif path:
        start = int(m.group("start"))
        count = int(m.group("count") or 1)
        changed[path].update(range(start, start + count))

cov = json.load(open(".evidence/coverage.json"))
unowned = []
for path, lines in changed.items():
    contexts = cov["files"].get(path, {}).get("contexts", {})
    for ln in sorted(lines):
        if not contexts.get(str(ln)):
            unowned.append(f"{path}:{ln}")

print(json.dumps({"changed_lines": sum(map(len, changed.values())),
                  "lines_with_no_owning_test": unowned}, indent=2))
sys.exit(1 if unowned else 0)

Two caveats: the contexts key only exists when contexts were recorded, and its shape varies across coverage.py versions, so check the actual generated JSON. Brand-new lines in brand-new files have no context, so treat "new file" as "requires at least one executed test that imports this module" rather than automatically failing.

Step Three: Fixture Read Set Closure

A patch can fail in two directions here: rewriting a fixture to make tests pass, or rewriting a fixture without ever running the tests that read it. If you maintain a registry indexed by fixture name, both can be detected.

pytest --fixtures-per-test -q > .evidence/fixture_usage.txt

# .evidence/fixture_readsets.yaml — human-maintained, not auto-generated
# Only list fixtures whose on-disk inputs are stable enough to hash.
db_session:
  reads:
    - tests/fixtures/schema.sql
    - tests/fixtures/seed_users.json
clock:
  reads:
    - tests/fixtures/clock.json

Closure rule: if the patch touches any file in a fixture's reads list, then every test declaring that fixture (per fixture_usage.txt) must appear in executed.jsonl. Otherwise the gate blocks, and the patch author must write an entry in .evidence/scope_exceptions.yaml noting the test name, reason, and expiry date. Exceptions must be reviewed, not silently passed through.

Step Four: Freeze the Flaky Quarantine Ledger

This is the highest-value gate, because a pressured agent has a cheap escape hatch: marking failing tests as skipped. Quarantine is reasonable — flaky tests destroy signal — but the list cannot be edited by the patch it protects.

**How to implement the quarantine hash-freeze mechanism**: In CI, computing a content hash of `.evidence/quarantine.yaml` (e.g., `sha256sum`) and comparing it against the hash from the previous commit on the baseline branch is the lowest-cost part of this gate to implement. The crux of the policy design is "direction of edit," not "the edit itself": adding an entry or extending `expires` gives the patch more tests it can skip, and must be blocked with a human sign-off required; while removing an entry (after several consecutive green runs) or merely editing `reason`/`tracker` does not affect the size of the quarantine set and can be auto-approved. Once the hash is written into a CI step, the agent cannot make failing tests "legally disappear" by quietly expanding the quarantine list in the same commit. If the team already has an ISSUE tracking system, you can require the `tracker` field to point to a real ticket and have the expiry date trigger a reminder, preventing the quarantine ledger from ballooning into a permanent signal black hole.
# .evidence/quarantine.yaml — content-hashed in CI
- nodeid: tests/test_ingest.py::test_retry_budget
  reason: timing-dependent under parallel execution
  added: 2026-09-02
  expires: 2026-11-02
  tracker: ISSUE-4412

Ledger change	Policy
Add a quarantined test	Block; needs human sign-off
Extend an existing expires	Block; needs human sign-off
Remove an entry after N consecutive green runs	Auto-allow
Edit only reason or tracker	Auto-allow

Hash the file in CI and compare against the hash stored on the baseline branch. Additions fail the build; removals pass. This single rule eliminates the most common way agent patches hide the regressions they introduce.

Step Five: Package the Evidence (Optional Second Reader)

All the steps above write to files, so the evidence bundle is mechanical:

File	Producer	Gate it serves
collected.jsonl, executed.jsonl	Recorder plugin	Inclusion
coverage.json	Coverage contexts	Reachability
fixture_usage.txt	pytest CLI	Closure
quarantine.yaml + hash	CI	Freeze

Once the evidence bundle is ready, a second model can act as an adversarial reader, reviewing the JSON rather than prose. Ask it fixed questions — "list every changed line with no owning test" "if this line is rolled back, which executed test would fail?" — and treat its answers as leads, not verdicts.

Running this review step in a clean environment requires a model that isn't billed per token. MonkeyCode's free model access and free server option are a low-commitment way to run audits on a temporary clone; free limits change over time, so check current terms before relying on it. Disclosure: this article was prepared as part of MonkeyCode product outreach.

Limitations and Non-Applicable Scenarios

• Reachability is line-level. A changed line within a covered region may still be logically untested; the gate narrows the search and does not draw conclusions.
• Selection auditing says nothing about whether the assertions are meaningful, nor whether the collection manifest has shrunk. Pair it with a suite manifest diff and falsifiability checks.
• Per-test contexts add runtime and storage overhead. On large suites, record contexts only for changed test files.
• If a full suite run takes only a few minutes, skip this audit and just run everything, skipping selection entirely. Auditing is only worth it where selection exists.
• If no one owns the fixture registry, skip it. A stale fixture_readsets.yaml blocks patches for the wrong reasons, and teams learn to bypass the crying-wolf gate.

The cheapest gate to add this week is the quarantine hash, because it needs no coverage tooling, just a CI step. The inclusion check comes next. Reachability and fixture closure are only worth the investment after the first two become boring and automatic.

One-line Summary

An AI agent's reported "42 passed" only proves the selected tests passed, not that the full suite of 900 tests passed.

Paragraph Summary

When an AI agent's patch reports "42 passed," it only proves those 42 tests ran; the 858 tests in the repository that were never executed are just unpriced assumptions. To address this, the article proposes an auditable evidence bundle with four gates: record the three counts collected, selected, and executed separately and perform an inclusion check (executed ⊆ selected ⊆ collected); use per-test coverage contexts to map changed lines to owning tests, finding unreachable changes with no owning test; maintain a fixture read-set registry for closure checking; and hash-freeze the flaky quarantine ledger, requiring human sign-off for new quarantines or expires extensions. The article also provides reference implementations such as a conftest.py recorder and reachability.py, packages the evidence for adversarial review by a second model, and finally discusses the limitations of line-level reachability and non-applicable scenarios.