Agent Patch Tests Don't Cover Changed Lines? A Diff Coverage Gate Solution

Title

Agent Patch Tests Don't Cover Changed Lines? A Diff Coverage Gate Solution

Meta Description

An Agent-generated patch passing tests doesn't mean tests covered the changed lines. This article introduces a diff coverage gate: intersect patch changed lines with test-executed lines, adjudicate by hunk as EXECUTED/PARTIAL/SILENT, reject merging untested code, with full Python implementation and CI integration.

Article Content

An AI Agent-generated code patch passing the test suite only means the run didn't crash—it doesn't mean the tests executed the lines the patch changed. This blind spot is significantly amplified in Agent programming scenarios: the model may add a defensive except branch, loosen type checks, or insert an early return that was never tested. File-level coverage is an average—a 91% file coverage can completely mask a brand-new unexecuted except branch. This article introduces a lightweight diff coverage gate: intersect patch changed lines with the lines tests actually executed, adjudicate each hunk individually, and reject merging code blocks that nobody executed.

Why File-Level Coverage Isn't Enough

File coverage is a statistical average. When a file shows 91% coverage, it may contain a completely unexecuted except branch, and the number barely moves. Agents happen to excel at generating this shape of patch: a small piece of defensive logic, a broadened type check, an early return the model imagined but never verified.

The question you actually need to answer during review is: did any test execute the lines I'm about to approve? Measuring line execution scoped to the patch is the smallest unit that answers this question.

Collecting Changed Line Ranges

The first step is to extract the added line number ranges of each hunk from the git diff. Three key parameters work together:

• -U0 keeps hunks minimal, avoiding one changed line dragging in 40 lines of untouched context
• --diff-filter=d skips deleted files, which have no executable lines
• The three-dot range base...HEAD diffs against the merge base rather than comparing to a drifted main branch

import re
import subprocess

def changed_ranges(base: str) -> dict[str, list[tuple[int, int]]]:
    diff = subprocess.run(
        ["git", "diff", "-U0", "--no-color", "--diff-filter=d", f"{base}...HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout
    ranges: dict[str, list[tuple[int, int]]] = {}
    path = None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            path = line[len("+++ b/"):]
            ranges.setdefault(path, [])
        elif line.startswith("@@") and path:
            m = re.search(r"\+(\d+)(?:,(\d+))?", line)
            start, count = int(m.group(1)), int(m.group(2) or 1)
            if count:  # pure deletions produce zero-length hunks; skip
                ranges[path].append((start, start + count - 1))
    return ranges

Pure deletions produce zero-length hunks, so they're filtered out; renamed files land under their new paths, aligning with coverage report paths. The output structure is file -> [(start line, end line), ...], cheap enough to recompute on every push.

Merge base is the most recent common ancestor commit of two branches, computed by the git merge-base command. When using the three-dot range base...HEAD, Git compares the diff between merge-base(base, HEAD) and HEAD, not the diff between the current tip of the base branch and HEAD. The reason: when base (e.g., main) advances after a feature branch is created, the two-dot range base..HEAD would include new commits on main in the diff, producing many changed lines unrelated to this patch and polluting the gate's adjudication scope. The three-dot range focuses only on changes introduced by the feature branch itself, making it more precise diff semantics for code review scenarios.

Collecting Executed Lines

Run the tests once with a coverage tool in branch mode, scoped to the source directory:

python -m coverage run --branch --source=src -m pytest -q tests/
python -m coverage json -o coverage.json

coverage json outputs executed_lines and missing_lines for each file. Omitting --source causes the intersection to be paid for by conftest.py, fixtures, and unchanged test helper files. Branch coverage isn't enabled by default in all toolchains—if disabled, an if that only runs one side will report as fully covered.

Branch coverage complements line coverage: line coverage only records whether a line was executed, while branch coverage also records whether each branch of a conditional statement (e.g., both true and false sides of if, triggered and untriggered except) was reached. In Agent programming scenarios, the defensive except branches models add often execute only on exception paths—line coverage may mark the whole structure as partially covered because the try block executed, but branch coverage clearly exposes that the except body never ran. With coverage run --branch enabled, coverage json output additionally includes executed_branches and missing_branches fields, which the diff gate can use to make stricter adjudications per hunk.

Intersection Classification and Adjudication

Each hunk gets one and only one verdict, and the gate acts on verdicts, not percentages.

import fnmatch
import json
import sys

SEVERITY = ["EXECUTED", "ALLOWED", "UNKNOWN", "PARTIAL", "SILENT"]

def classify(path, span, ran, allowlist):
    lines = set(range(span[0], span[1] + 1))
    if any(fnmatch.fnmatch(f"{path}:{span[0]}", pat) for pat in allowlist):
        return "ALLOWED"
    hit = lines & ran
    if not hit:
        return "SILENT"
    return "EXECUTED" if hit == lines else "PARTIAL"

The five verdicts and their default merge rules:

Verdict	Meaning	Default merge rule
EXECUTED	All changed lines were executed by some test	Pass; check assertions
PARTIAL	Some changed lines never executed	Block merge
SILENT	No changed lines executed	Block merge
UNKNOWN	Run excluded some tests; result indeterminate	Rerun the excluded subset
ALLOWED	Explicit, owned, reviewed exception	Pass, with owner recorded

A patch that adds a retry branch but never tests it, sample output:

EXECUTED  src/ledger/charge.py:41-48
PARTIAL   src/ledger/charge.py:52-61
SILENT    src/ledger/charge.py:64-70

1 hunk(s) at or above PARTIAL

Hunk is a contiguous region of changes in unified diff format, identified by a @@ ... @@ header. Each hunk contains some context lines, deleted lines (starting with -), and added lines (starting with +). In -U0 mode, context lines are compressed to zero, so a hunk contains only the actually added/deleted lines, and each hunk's added line number range directly corresponds to the executable code interval introduced by the patch. The point of adjudicating per hunk rather than per file: a single file may contain both tested and untested changes, and file-level pass/fail loses this granularity; hunk-level adjudication lets the gate pinpoint exactly "which block of added logic wasn't reached by any test," and reviewers can navigate directly to the specific line numbers.

Setting Up the Gate in CI

Severity ordering is the core of the whole enum design. --fail-on PARTIAL blocks both PARTIAL and SILENT; --fail-on SILENT blocks only hunks that were not touched at all. We recommend running strict mode for one sprint cycle first, then deciding whether PARTIAL is noise or real signal in your codebase. Use exit code 2 so it's distinguishable from test run crashes (exit code 1) in most CI systems.

python diffcov.py --base origin/main --fail-on PARTIAL --excluded-tests 0

What Each Verdict Can't Prove

EXECUTED proves a line was executed, but not that any test asserted the line's effect. A test that only calls a function and checks "didn't raise" will mark every internal line as EXECUTED. Treat verdicts as permission to read the assertions, not as evidence they exist.

PARTIAL is often the most informative line: a branch only took one side, or a loop body was only entered by an empty fixture. ALLOWED is a declaration rather than a fact—each pattern needs an owner and a justification, and is reviewed like code. Keep patterns line-scoped; something like src/migrations/* lets the allowlist turn the gate into decoration.

Toolchain limitations are real: C extensions report nothing, subprocess code reports nothing, and async task bodies may execute after coverage is turned off. Code that only runs on real devices will forever show SILENT.

Flaky tests are tests whose results are inconsistent across multiple runs with the same code and environment—sometimes passing, sometimes failing, usually caused by timing dependencies, shared state, network, or random data. In the context of coverage gating, the isolation strategy for flaky tests directly affects the credibility of verdicts: excluded tests no longer execute, their covered lines appear unexecuted in this run, downgrading hunks that would have been EXECUTED to SILENT. If the gate doesn't distinguish "code really wasn't tested" from "test was temporarily excluded," it produces many false positives, forcing teams to loosen thresholds or bypass the gate. The --excluded-tests parameter's job is precisely to model this uncertainty explicitly as UNKNOWN, requiring a full rerun of the excluded subset before merge, rather than letting an isolation strategy quietly erode the gate's authority.

Interaction with Flaky Test Isolation

When a flaky test is excluded from the run set, coverage silently drops. A hunk originally covered only by that test now shows SILENT—the gate isn't wrong, the run is incomplete. This is why --excluded-tests downgrades SILENT to UNKNOWN instead of failing outright: an honest "I don't know" beats a false block, which teaches reviewers to bypass the gate.

Pass the exclusion count explicitly from the test runner, and require a full rerun of the isolated subset before merge. The inverse failure is worse: broad isolation must never cause UNKNOWN to become the default. If more than a small number of hunks return UNKNOWN, you don't have a coverage run—you have a habit of skipping tests.

Seeding the Gate and Tool Choices

A gate that has never seen a failure is just decoration. You need a stream of patches—some correct, some containing untested branches—to validate that it blocks what it should and passes what it should. Generating candidate patches with a free model tier is a legitimate use: ask for a behavior change plus tests, then just run the gate and record the verdicts that appear. At this point the model is a fixture generator, not an oracle; weak samples only produce easy EXECUTED lines, which is itself a useful calibration point.

MonkeyCode's free model access and free server option map neatly to the two logistical ends of this loop: candidate generation and clean-checkout execution. Coverage tasks must run each candidate on a fresh checkout with the same toolchain as CI, otherwise the gate gets skipped exactly when the queue is busy. Two constraints to honor: treat the runner as ephemeral (regenerate coverage.json each run, store nothing non-reproducible), and pin toolchain versions (runtime, coverage tool, and test runner), because verdicts from different versions aren't comparable to CI results.

When Not to Use This

• Repos where most changed lines are generated code: lock files, protobuf output, migration files. The allowlist swallows the gate, and reports go unread.
• Toolchains for the tested language without line coverage support. Partial instrumentation produces confident but wrong SILENT lines.
• Teams with flaky rates so high that most runs end with partial coverage. Fix the isolation budget first; this gate amplifies existing noise.
• People who intend to treat the number as a quality metric. It's a merge decision about a diff, not a score for a person or a repo.

The script is under ninety lines and runs in seconds, because the most expensive part—the test run—is already happening. What changes is that the report now explicitly lists the exact lines the Agent changed but didn't execute, instead of hiding them behind a green checkmark.

One-line Summary

Use a diff coverage gate to adjudicate by hunk whether an Agent patch's changed lines were executed by tests, rejecting merges of unexecuted code.

Paragraph Summary

The article notes that file-level coverage is a statistical average and can't answer the key question of whether an Agent-generated patch's changed lines were executed by any test. The author proposes a lightweight diff coverage gate: extract each hunk's added line number range from the git diff, intersect with coverage.py's executed_lines, output one of five verdicts per hunk—EXECUTED, PARTIAL, SILENT, UNKNOWN, ALLOWED—and decide whether to block merging. In CI, the gate controls the failure threshold via severity ordering, uses --excluded-tests to downgrade flaky-isolation-induced SILENT to UNKNOWN to avoid false blocks. The article also discusses each verdict's proof boundaries, toolchain limitations, how to seed the gate, and when not to use it.