Six Fixes to a Merge Gate's Failure: Lessons on the Code Verification Dilemma for AI Coding Agents

AI coding agents generate patches — can they be merged? This question looks simple, but anyone who has actually worked on a merge gate will tell you: the phrase "tests passed" may be far less reliable than you think.

Developer Sanflow10 recently published a detailed retrospective documenting his experience building AdversaryGate, a merge gate tool for AI coding agents, which shipped six patch versions in four days. The core conclusion: almost all vulnerabilities stem from the same pattern — fail-open by omission. Certain inputs fall outside the gate's measurement scope, the gate silently skips them without reporting, and the rest of the patch cruises through to green light on that "passed" conclusion.

The Design Intent Behind AdversaryGate

AdversaryGate's core logic is not complicated: run the test suite on the baseline code and the patch separately, then issue one of three verdicts — MERGE (exit code 0), BLOCK (exit code 1), or INCONCLUSIVE (exit code 2).

The single design principle underpinning the entire tool is one sentence: A measurement nobody actually performed cannot be used to pass a gate. It sounds self-evident, but in practice, loopholes are everywhere.

Mutation Testing is the key background for understanding several vulnerabilities in this article. The basic idea is: inject small, artificial defects (called "mutants") into the code under test, then run the test suite to see whether the tests catch these defects. If a test fails, the mutant is said to be "killed," indicating that the test actually constrains that piece of code. If the test still passes, the mutant "survives," meaning the test has a blind spot. The mutation score is the proportion of killed mutants out of all mutants, often used as a quantitative measure of test quality. AdversaryGate borrows this mechanism to assess the test sufficiency of patches. However, if the mutant generation strategy itself is flawed — for instance, only mutating the first 12 changed lines, or failing to cover same-line branches — then a high mutation score can be an illusion: the tests don't truly cover all changes, yet they show a perfect score because no mutants were generated. This is one of the root causes of the fail-opens described repeatedly in this article.

Fail-open (failure means allow) is a security design pattern opposed to fail-safe. In a security system, when a component errors out or cannot make a determination, fail-open means the system defaults to allowing the operation to proceed, while fail-safe defaults to denial. In AdversaryGate's verdict logic, MERGE represents allowing, and INCONCLUSIVE represents an inability to determine. The so-called "fail-open by omission" refers to: when some input falls outside the measurement scope, the gate does not flag it as INCONCLUSIVE but simply ignores the missing input, allowing the "passed" conclusion from the remaining measurements to carry over, and the patch thereby receives MERGE. This pattern runs through all vulnerabilities in this article: binary markers hiding changed lines, documentation masquerading as code, test-named modules evading detection — all are essentially "no measurement" being treated as "measurement passed."

Six Ways to "Fool the Gate"

During iterations from 3.0.0 to 3.0.6, Sanflow10 actively attacked his own gate, discovering one by one various methods that let patches which shouldn't pass obtain a MERGE.

Tests that read code instead of running it

AG-046 (fixed in 3.0.0): A test that hashes the changed file will fail under any edit, so it "kills" all mutants, yielding a mutation score of exactly 1.0. The fix introduces a "comment canary" — appending a comment to the end of changed lines then re-running the assertions. If an assertion fails because of this, it means the assertion is reading text rather than verifying behavior, and that score is marked as unmeasured.

Comment Canary is an active probing mechanism designed by AdversaryGate to identify "pseudo-tests." Its principle: insert a harmless comment (canary) at the end of a changed code line, then re-run the relevant test assertions. If a test assertion fails after this modification, it means the assertion is actually reading code text content (such as hashes or string comparisons) rather than verifying runtime behavior — because comments do not change program semantics, and genuine behavioral tests should not be affected by them. Such tests are marked as "unmeasured," and their mutation score no longer counts toward a valid rating. The canary mechanism embodies the design philosophy of "actively attacking your own measurement tool": instead of assuming a test is a test, it introduces controlled perturbations to verify the test's true nature. Similarly, version 3.0.1 added no-op assignments to the canary to perturb both AST (abstract syntax tree) and bytecode-level comparisons, preventing assertions that only compare syntactic structures from bypassing detection. But the author honestly admits that if assertions only read smaller fragments of the syntax tree, they may still escape, illustrating that the boundaries of a measurement tool are recursive.

The canary itself can be bypassed

3.0.1 discovered that assertions comparing AST or bytecode can still pass the comment canary. The canary then added no-op assignments to alter the syntax tree and bytecode. The author candidly acknowledges this is a partial fix: if the assertion only reads smaller fragments of the syntax tree, it can still pass. This is documented honestly in the README.

Binary markers in .gitattributes

Another vulnerability fixed in 3.0.1: A .py file marked as binary in .gitattributes causes git diff to output Binary files ... differ, the changed lines vanish from the coverage diff, and the patch immediately enters MERGE. Now such diffs are rejected.

Documentation changes get a perfect coverage score

Deleting one line of README can make coverage hit 1.0 — because documentation is treated as code. After the fix, patches with no code lines have their coverage marked as unmeasured, and cannot earn a MERGE on the basis of coverage.

Code Coverage in this article's context specifically refers to the proportion of code lines or branches actually reached when the test suite executes. Common metrics like line coverage and branch coverage are widely used to judge test sufficiency, but their fatal weakness is: coverage only measures "which code was executed," not "whether the executed code was correctly asserted." If a test calls a line of code but does not verify its output or side effects, coverage still rises. An even more extreme case is when the coverage tool mistakenly counts non-code files (such as README) as executable code — deleting a line of documentation may shrink the denominator, passively pushing coverage to a perfect score. AdversaryGate's fix for this: coverage is only calculated when the patch contains code lines; patches with no code lines have coverage marked as "unmeasured," thereby cutting off the path of obtaining MERGE by inflating coverage through documentation changes. This once again confirms the core principle — an unmeasured metric cannot serve as grounds for passing.

Same-line branches and insufficient mutation coverage

3.0.4 fixed two problems: first, same-line branches like return 999 if x == 0 else x + 1 could still reach MERGE with 5/5 mutants killed; now the gate generates branch mutants, and survivors become blind spots that trigger INCONCLUSIVE. Second, previously only the first 12 changed lines were mutated, so subsequent untested changes would never be broken; now every changed line is broken at least once.

Test-named product modules

AG-047 (fixed in 3.0.6): The gate determined what is a test solely by path. A module named like a test but imported by the code under test was excluded from coverage and mutation, and changes to it were never judged. Now test-named modules imported by non-test code are treated as source code. The author honestly notes: this fix has been verified at the mechanism level but not reproduced as an end-to-end attack; modules dynamically imported via importlib or __import__ still cannot be detected.

Not Just Fail-open: False Positives Are Also Happening

Not all problems were about letting through things that shouldn't pass. Some were the opposite — honest patches wrongly judged as INCONCLUSIVE. The \ No newline at end of file marker was treated as a line, causing line-number offsets; MCP's 900-second timeout was shorter than the full test suite's runtime; on Android (Termux), the default memory limit caused every process to be aborted. These were fixed in 3.0.5.

Hardening the Release Process

Versions 3.0.2 and 3.0.3 were released with API tokens, the MCP package remained at 3.0.0, and core files lacked attestation. The author removed all versions prior to 3.0.4 from PyPI and GitHub. The release process is now unidirectional: signed tag → Trusted Publishing → both packages carry PyPI attestation.

Known Limitations

AdversaryGate still has clear boundaries: measurement only supports Python and pytest, all other languages return INCONCLUSIVE rather than a speculative MERGE; it cannot prevent an agent running as the same OS user from accessing the gate's temporary directory; the canary fixes are partial, and each limitation is documented right next to its fix.

Every discovered vulnerability is recorded in the public ERRORS_AND_INCONSISTENCIES.md, including reproduction steps and fixes.

Attestation is a key concept in software supply chain security, referring to verifiable metadata generated by a build system to prove that an artifact was indeed built from specific source code, in a specific environment, through a specific process. In the Python ecosystem, PyPI supports attestation based on Trusted Publishing, where the build process is issued short-lived credentials by a CI/CD platform (such as GitHub Actions) without long-term API tokens. The value of attestation lies in: even if an attacker gains publishing privileges, it is difficult to forge a chain of proof consistent with the legitimate build process. In this article, versions 3.0.2 and 3.0.3 were released with long-term API tokens, posing a supply chain risk — an attacker who steals the token could publish a malicious version. After switching to Trusted Publishing, publishing credentials are bound to a specific CI workflow, and both packages are accompanied by attestation, significantly improving non-repudiation and auditability of releases. This fix is consistent with the security logic of the gate itself: not only must patches be verified, but the process of releasing patches must also be verified.

Implications for the AI Coding Tool Ecosystem

The value of this case extends beyond a single tool. As AI coding agents increasingly generate and submit code, "how to verify an agent's output" is becoming an independent and urgent problem. AdversaryGate's experience reveals a deep contradiction: verification tools themselves also need to be verified, and the boundaries of verification are often blurry.

Sanflow10's approach is worth emulating — actively attacking his own tool, publicly documenting all failure cases, and annotating known limitations next to each fix. This attitude of "writing down what is unsafe" is far more valuable than claiming "our gate is secure."

If you can make 3.0.6 produce a false MERGE, the author says he'd very much like to know.

One-line Summary

A four-day, six-version retrospective on an AI coding agent merge gate: nearly all vulnerabilities stem from 'fail-open by omission.'

Paragraph Summary

Developer Sanflow10 built AdversaryGate, a merge gate for AI coding agents, and shipped six patch versions in four days. Its core principle is 'an unmeasured metric cannot be used to pass a gate,' yet nearly all vulnerabilities stem from the same pattern — fail-open by omission: certain inputs fall outside the measurement scope, the gate silently skips them, and the patch earns MERGE on the 'passed' conclusion of the remaining parts. The article details six ways to fool the gate, including tests that read code instead of running it, the comment canary being bypassed, binary markers hiding changed lines, documentation masquerading as code, missed same-line branch mutations, and test-named modules evading detection. There are also cases of false positives where honest patches were misjudged as INCONCLUSIVE. The author actively attacks his own tool, publicly documents every failure case and known limitation, and changed the release process from token-based publishing to signed tags plus Trusted Publishing.