AI-Generated Tests Hit 100% Coverage but Miss 96% of Bugs

One team ran mutation testing on an AI-generated test suite. The report said 100% line coverage — every line of code had been executed. But the mutation score — the share of deliberately injected bugs the tests actually caught — was 4%. In other words, 96% of injected defects strolled right past a suite that claimed "full coverage."

That number is far sharper than the old line about "AI hallucinates." Coverage tells you code ran. It tells you nothing about whether anything was checked.

Why coverage lies

The way line coverage is measured has a built-in loophole: a statement counts as "tested" as long as it was executed inside a test function — even if the assertion that follows is expect(result).not.toBeNull() on a function that could never return null.

Thoughtworks' Technology Radar called out this problem long ago: high coverage numbers routinely mask logically empty tests, and AI-generated tests are especially prone to it.

There's a structural reason behind this. When the same model writes both the implementation and the tests, both inherit the same blind spots. A practitioner on HackerNoon described a scenario like this: a counter that incremented regardless of whether the request succeeded, while the test suite treated "the counter went up" as evidence the request succeeded — because that assumption was baked into the very code it was mimicking. The bug and the test nodded at each other, nobody objected, so no test ever turned red.

Mutation testing is the reverse check: inject a small fault (flip a comparison operator, delete a line, change a constant), then re-run the tests. If no test fails, the mutant has "survived" — and a surviving mutant means some class of real bug can ship under the cover of an all-green suite. In one informal report, 20 AI tests written against a Laravel controller claimed to cover every endpoint, yet killed only 68 of 95 injected mutants — 27 bugs slipped past "complete coverage."

Mutation Testing is a technique for evaluating the effectiveness of a test suite. Its core idea is to deliberately inject tiny artificial defects into the source code — called "mutants" — such as flipping a comparison operator, deleting a line, or replacing a constant — and then re-run the tests. If a mutant causes at least one test to fail, the mutation was killed by the tests, meaning the tests genuinely exercised the relevant logic; if all tests still pass, the mutant has survived, meaning the suite is completely blind to that class of real bug. The mutation score is the proportion of killed mutants out of the total injected. Whereas line coverage only cares whether code was executed, the mutation score cares whether the code's correctness was verified by assertions. It essentially simulates whether real bugs would be caught by the tests, which is why it's considered a stricter standard for test-suite quality — at the cost of higher computation, since every mutant requires re-running the tests.

Line Coverage is one of the most common code coverage metrics, measuring the proportion of source lines that were executed at least once during a test run. For example, an if statement counts as covered as long as it was entered once, regardless of whether the test checked the outcome of the condition. Similarly, once a function is called, every line in its body counts toward coverage, even if its return value is never asserted. High line coverage therefore guarantees nothing about whether the tests checked any logic — it only proves the code ran. More refined metrics include branch coverage (checking whether every conditional branch was executed), path coverage (checking all possible combinations of execution paths), and condition coverage (checking the truth-value combinations of each sub-condition in a compound condition) — but even these still can't substitute for validating that assertions are meaningful.

A more extreme example: a check that is structurally incapable of failing

A reader described something in the comments that's worse than a weak assertion: a check that is structurally incapable of failing.

Their example: an automated mobile check hashes screenshots before and after a swipe to confirm the feed advanced. But if a video is playing on screen, the hash changes every frame — so the check passes whether or not the swipe worked. A second check reads process uptime via ps -o etimes=, but that command doesn't exist on macOS at all; it fails silently, uptime never crosses the threshold, and the rule never fires. Neither check ever turned red, so neither was ever sent for review.

The question they posed is worth copying verbatim: If the thing I care about didn't happen, would this check still pass?

That's mutation testing, one-sentence version — done by hand instead of with a tool.

False Positive refers to a test reporting failure when there's actually no bug, while False Negative refers to a test reporting success when a bug genuinely exists — and the "structurally incapable of failing check" described in the article is an extreme form of the false negative. The defining trait of such checks is that there is no valid connection between their passing condition and the goal they claim to verify. For example, comparing screenshot hashes of dynamic content (like a playing video): the hash is always changing, so the comparison always comes out "different" regardless of whether the swipe worked — and the check always passes. Or relying on a command that doesn't exist on some platforms (like ps -o etimes= on macOS): the command fails silently, the script never errors out, and the check never triggers. This kind of check is more dangerous than a weak assertion because it masquerades as valid verification, passes code review, yet responds to real failures with nothing but silence. The way to spot it is the question the article proposes: if the thing I care about didn't happen, would this check still pass?

What you can do right now

You don't need to install a mutation testing framework to get started. Before trusting any AI-written test — whether you wrote it or a colleague did — just ask the question: if I introduced the bug this test is supposed to catch, would it fail? If the answer requires running it to find out, that itself is a review item, not "did it pass?"

The author maintains a free experimental repo that turns this contrast into something you can run. node run-lab.js weak generates AI tests against deliberately broken code — all three tests pass. node run-lab.js verify-candidate then runs the same code against an independent verification suite — it fails, and catches all four seeded bugs. The gap between those two runs is the entire argument of this article, reproducible in about two minutes.

Key takeaway

Coverage measures whether code was executed; the mutation score measures whether the tests actually check anything. The gap between them is exactly where AI-generated tests most easily fool people — when the same model writes both the implementation and the tests, their blind spots align perfectly, so no test ever lights up red.

Judging whether a test is trustworthy doesn't require building a toolchain first. Asking "if I introduced this bug, would it fail?" filters out most paper-thin tests.

One-line Summary

AI-generated test suites often report 100% coverage but score only 4% on mutation testing — coverage proves code ran, not that logic was checked.

Paragraph Summary

Through a mutation testing case study of an AI-generated suite, the article reveals that under 100% line coverage the mutation score was just 4%, with 96% of injected bugs escaping detection. The reason: line coverage only counts whether code was executed, entirely ignoring whether assertions are meaningful. Worse, when the same model writes both implementation and tests, their blind spots align perfectly, causing tests to treat wrong assumptions as correct evidence. The article also describes "structurally incapable of failing checks" — such as hashing screenshots of a playing video or relying on a nonexistent command — which always pass and are never reviewed. The author's core test: if I introduced this bug, would it fail? No toolchain required to filter out most paper-thin tests.