Why AI Coding Agents Shouldn't Test Their Own Code

A Counterintuitive Research Finding

Today's AI coding agents — whether Devin, Cursor, or Factory — can write code, run tests, and automatically fix issues. But a new study has surfaced a finding that cuts against conventional wisdom: preventing AI coding agents from writing their own tests actually makes them perform better.

The numbers are specific: when coding agents stop generating their own tests, task success rates improve, overall speed increases by roughly 6%, and costs drop by around 9%. This might sound like taking a step backward — isn't more testing always better? But the key issue isn't the testing itself; it's who does the testing.

The Core Problem: The Builder Can't Also Be the Inspector

The underlying logic is straightforward. When a coding agent writes code, it does so based on its own interpretation of the requirements. When that same agent then writes tests, it's still working from that same interpretation.

The result: the tests it writes will almost inevitably pass the code it wrote. Because both stem from the same mental model, the tests introduce no external perspective to question whether the implementation actually matches what the user intended.

This means you're spending compute and time running tests without getting any meaningful validation — the tests pass by default, rendering them useless. You're paying for a self-congratulatory performance.

The Fundamental Flaw of Self-Testing

This isn't a matter of the agent lacking capability — it's a structural conflict of interest. In traditional software engineering, having developers write unit tests for their own code is certainly valuable, but those are human engineers. Humans make "comprehension errors," and tests are partly meant to surface those errors.

But when an AI agent is both the builder and the inspector, its "comprehension errors" seep into both the code and the tests simultaneously. If it misread the requirements, its tests will validate along that same misreading and "pass" cleanly — masking the real problem.

The takeaway isn't to skip testing. It's to stop letting the builder be the only inspector.

This problem has a well-known epistemic root in software engineering: Confirmation Bias. Human engineers writing tests also tend to unconsciously verify only "the paths they believe are correct" — which is exactly why many teams adopt Test-Driven Development (TDD), writing tests before implementation to force developers to define "what correct looks like" before touching any code. But TDD fails equally for AI agents: even if you have the agent write tests first, it's still using the same language model within the same context window to interpret requirements, making it impossible to truly decouple the "question setter" from the "answer writer." This is why the solution must introduce separation at the system architecture level, rather than simply reordering the prompts fed to a single agent.

One Solution: Let an Independent Agent Test the Real Application

The video introduces a tool called Kane CLI that addresses this problem directly. Its approach is to strip the testing phase away from the coding agent and hand it to an independent testing agent.

Kane CLI runs directly in your terminal. You describe a user flow in natural language, it executes that flow in a real browser, and then feeds any failures back to your coding agent for fixes.

Isn't This Still AI Writing Tests?

This is the most natural pushback: if another AI is doing the testing, isn't it still "AI writing tests"?

Yes — it is AI doing the testing. But the difference lies in division of responsibility and perspective:

• The coding agent writes code and cares about implementation logic;
• The testing agent validates the running application and cares about what users actually see and experience.

The two agents don't share the same understanding of the codebase. The testing agent operates from a "user perspective," running complete flows in a real browser and verifying the final rendered result — not whether the internal code logic is self-consistent. This introduces the external verification perspective the coding agent inherently lacks.

This "user-perspective validation" corresponds in engineering practice to End-to-End Testing (E2E), which sits at the top of the testing pyramid alongside unit tests and integration tests. E2E tests simulate real users operating in real environments, validating the complete system behavior from input to output — not the internal logic of any single function or module. Traditional E2E tools (like Selenium or Playwright) require engineers to manually write scripts describing user actions. Kane CLI's approach replaces those scripts with natural language, letting the testing agent translate natural language into actual browser action sequences. This allows non-technical stakeholders to participate in defining acceptance scenarios, while preserving E2E testing's core advantage of closely mirroring real user experience.

Implications for AI-Assisted Development Workflows

As coding agents become more deeply embedded in development pipelines, the question this research raises deserves serious attention from engineering teams. Blindly letting a single agent own the entire "write code → write tests → self-validate" chain likely produces nothing more than the illusion of having been tested.

A more robust approach is to introduce separation of responsibilities into the workflow:

• Code generation and quality validation are handled by different agents;
• Tests are anchored as close to the real user environment as possible (e.g., end-to-end flows in a real browser), rather than staying at the code-unit level;
• Test results feed back to the coding agent as independent signals, forming a closed-loop repair cycle.

This architecture avoids the blind spots of self-validation while delivering unexpected gains in cost and speed — because once ineffective tests are eliminated, the agent stops wasting resources on meaningless verification.

Ultimately, the reliability of AI-assisted development doesn't depend on how capable any single agent is. It depends on whether the overall system contains genuinely independent checkpoints. Developers who want to explore this approach can try the relevant tools at testmuai.com.