Are 90% of AI-Written Tests Pseudo-Logic? Three Blind Spots and Probe Tactics

AI-generated code is becoming increasingly common, but can AI-written tests really hold the line on quality? A tech talk on Bilibili delivered a rather uncomfortable conclusion: a large number of AI-generated unit tests only cover the happy path, while negative and boundary cases are almost entirely missed. More critically, the author found through hands-on testing that roughly 90% of these tests still pass after mutation—meaning the tests never actually "locked in" the logic; they just looked like they passed.

Why "Power Tests" Are Actually Dangerous

The video uses the term "power tests," referring to tests that pursue rapid generation and get merged as soon as they're written. AI's motivation for writing tests differs from that of human engineers: it tends to output shallow "input-assert output" structures, because these patterns are the most common in training data and the easiest to judge as "correct." As a result, side-effect states, exception branches, and boundary values—the very places where problems actually occur—become blind spots.

The danger is that coverage numbers create an illusion of "already tested." Most teams stop at the first two blind spots, believing quality is assured, when in fact the core logic was never verified at all.

Three Blind Spots: Fake Coverage, Hollow Logic, Missing Side Effects

The author breaks down the problems with AI tests into progressively deeper categories:

Fake coverage is the most surface-level issue—the tests run, coverage goes up, but the assertions themselves are weak, which amounts to not testing at all.

Hollow logic is more insidious. The test code has a complete structure, but the assertions don't actually validate business rules—for example, only checking that the return value isn't null, without checking whether it's correct.

Missing side effects is the one most likely to bite in production. When tests involve database writes, external API calls, or state changes, AI-generated tests often only verify return values without verifying whether those side effects occurred as expected.

There's another layer: the absence of boundary-value probes. This is the "final blow"—extreme inputs, empty strings, out-of-range parameters are exactly where real failures cluster.

Coverage typically refers to line coverage, branch coverage, and similar metrics, measured by tools like JaCoCo or Coverage.py. It answers "which code was executed," not "whether the execution results were correctly asserted." Code can be 100% executed with no meaningful assertions—that's the root of "fake coverage." Boundary value analysis is a classic black-box testing method: for each input range, pick the minimum, maximum, just below minimum, just above maximum, empty value, None, and so on—because off-by-one errors, empty collections, and integer overflows almost always occur near boundaries. Parameterized tests (such as Pytest's @pytest.mark.parametrize) are the standard way to batch-hit these boundary points, covering large combinations of boundary probes at very low cost.

Once Mocks Cross the Line, Tests Lose Their Meaning

The video makes a spot-on observation: once mocks cross the line, tests go from verifying business logic to verifying mocks. When the subject under test depends on a database or external API, if the probes never actually hit those external dependencies, high coverage is just self-deception.

This also explains why "all-mock" tests looked great under the old approach but need to be re-examined under the new one.

Mocks in unit testing refer to substitute objects that replace real dependencies (databases, HTTP clients, message queues, etc.), allowing tests to run independently of external environments. Moderate mocking isolates uncertainty, but once all external interactions the subject depends on are replaced, the test is really only verifying "whether the code under test called the mocks as expected," not "whether behavior is correct under real dependencies." This has two consequences: first, assertions easily degrade into checks on mock call arguments, leaving business rules unverified; second, real-environment issues like database constraints, serialization formats, and network timeouts can't surface at all. The industry typically balances this with the "testing trophy": few but focused unit tests, integration tests covering real dependencies, and end-to-end tests as a safety net for critical paths—avoiding an inverted test pyramid that turns into an "ice cream cone."

Hands-On: Four Probe Weapons

On the methodology side, the author provides a concrete tool combination, in order of priority:

1. Mutation testing: Run mutations with P-Test or Cosmo and observe the mutation pass rate. This is the core means of judging whether tests truly constrain the code.
2. Boundary probes: Use parameterized tests to specifically hit boundary inputs like None, empty strings, and extreme values.
3. Coverage: Use as a fallback reference, not as a quality metric.
4. Side-effect hooks: Prevent production from being bitten by test side effects.

You don't need many tools—the key is hitting the right spots. The author's criterion is direct: a mutation pass rate below 70% means the tests have holes and the probes aren't hitting their targets.

The implementation process can be summarized in a few commands: install dependencies, run mutations to check the pass rate, use Pytest to hit boundary markers individually. The process itself isn't complicated—the hard part is making it a habit.

Mutation Testing works by injecting tiny artificial defects into source code (e.g., changing > to >=, + to -, deleting a statement), generating a batch of "mutants," and then running the existing test suite against them. If a mutant causes tests to fail, the tests have "killed" that mutation, proving they constrain that piece of logic; if the mutant survives (tests still pass), the tests didn't cover the modified line. Mutation pass rate = surviving mutants / total mutants—the lower the number, the stronger the tests. P-Test and Cosmo are Bilibili's internal mutation testing platforms for Java/Go services, handling automatic mutant generation, test scheduling, and score aggregation. Compared to coverage, mutation score measures "whether assertions are really checking business rules," making it much harder to fool with fake coverage.

Old Approach vs New Approach: Four Contrasts

The author clarifies the differences with four comparisons:

• Coverage vs mutation score
• All-mock vs real probes
• Happy-path cases vs boundary cases
• Write and forget vs continuous mutation

The new approach doesn't overturn the old one—it adds a layer of probes on top. "Writing tests and feeling confident" versus "writing tests and running probes" are two fundamentally different quality mindsets. The former looks at coverage numbers; the latter looks at mutation scores—that's where the gap lies.

Wrapping Up in Three Sentences

The video closes with three core takeaways: coverage does not equal correctness; probes must penetrate to real dependencies; boundaries are the core battlefield.

Next time AI helps you generate tests, don't rush to merge. Run mutation testing and boundary probes first to see whether those tests truly lock in the logic or just look like they pass. The author also previews follow-up content on the loop of AI fixing bugs and how to make AI fix bugs without introducing new ones.

For engineering teams, the value of this thinking goes beyond AI-generated tests—any test process that emphasizes "fast output, fast merge" deserves a fresh look through probes.

One-line Summary

About 90% of AI-generated unit tests still pass after mutation—coverage is an illusion; rebuild the quality line with mutation testing and boundary probes.

Paragraph Summary

Based on a tech talk on Bilibili, the article highlights common pitfalls of AI-generated tests: many only cover the happy path, with weak assertions, missing boundaries, and unverified side effects. Roughly 90% of tests still pass after mutation, showing they don't truly lock in logic. The author breaks the problem into three blind spots—fake coverage, hollow logic, and missing side effects—and warns that excessive mocking turns tests from verifying business logic into verifying mocks. The methodology offers four probe weapons: mutation testing first, boundary probes next, coverage as a fallback, and side-effect hooks to protect production, with a mutation pass rate below 70% as the criterion for holes in tests. The article stresses that coverage does not equal correctness, and any fast-output, fast-merge test process deserves re-examination through probes.