Why Is Writing AGENTS.md First the Wrong Move When Handing Legacy Code to AI?

Title

Why Is Writing AGENTS.md First the Wrong Move When Handing Legacy Code to AI?

Meta Description

Handing legacy code to AI for refactoring? Writing AGENTS.md first is wrong. The real prerequisite is runnable tests as an exit condition—let AI draw the map itself. Drawing on Burn's migration of 530,000 lines of Zig, the METR experiment, and Google's migration process, this article explains why tests come first, how checks should be tiered, and how to use characterization tests as a fallback when legacy code has no tests.

Article Content

When handing a ten-year-old repository to AI for refactoring, most people's first instinct is to write a detailed AGENTS.md covering each module's responsibilities, architecture, and caveats. But a counterintuitive point shared by Kaolab on Bilibili is that the order should be reversed—give tests first, then the map. Because AI can draw the map itself, but it can't tell itself whether "the change is correct."

Tests Are the Exit Condition for AI Code Changes

In a summary of a large-scale code migration, Anthropic ranked "having a reliable judge first" ahead of all other steps. The reason is straightforward: Claude Code's official documentation states that it stops when it thinks the job is done. Without a runnable check, "looks done" is its only signal—this migration would have neither an exit condition nor a way to measure success.

A convincing counterexample: Burn handed 530,000 lines of Zig code to Claude to translate into over 1 million lines of Rust. Before merging, the test suite passed completely, including over 1 million assertions (tests were written in TypeScript, independent of the implementation language, so switching languages didn't require switching judges). Yet even with everything green, 19 subtle semantic regressions still surfaced after merging. Test coverage shows it didn't violate the portion of behavior written into the tests; the portion not written down is beyond the tests' reach.

Who Should Draw the Map

The map is certainly needed. The top success factor AIbin summarized when migrating test files was: picking out the correctly relevant files for it. The question is who should draw this map.

Claude Code's documentation specifically lists what AGENTS.md should not contain: file-by-file introductions to the codebase, and anything it can figure out by reading the code. What it should contain is what it can't guess—commands to run, project-specific architectural decisions, and non-obvious pitfalls. As for which files reference which, leave that to tools: Google uses code indexing to trace references outward layer by layer in internal code migrations, and in Anthropic's migration summary, dependencies were also mapped with deterministic scripts.

Why are the few lines written by hand actually more valuable? METR ran a randomized controlled experiment: 16 experienced open-source developers working in large repositories they had maintained for years, averaging over 1 million lines of code each. Before using AI, they predicted they'd be 24% faster; the actual result was 19% slower than without AI. One explanation the researchers gave is that such projects contain many implicit requirements—how docs are written, what level tests should cover, how formatting is arranged—that take humans a long time to learn. What's truly missing from the map is exactly this, and it's precisely what AI can't extract by reading code.

Checks Should Run from Cheap to Expensive

In Google's migration pipeline, every change made by the large model must pass checks in order from cheap to expensive: whitespace, syntax, parsing, compilation, and finally regression tests. If any stage fails, the change is discarded outright and handed to a human. Across 39 migrations and 595 code changes, 74% were generated by the large model. The paper also recorded a failure cause: some tests were already red, so the test stage couldn't even run—so tests aren't just for use; before handing off, you also need to make sure the main branch is green.

AIbin's approach was likewise driven by checks: 3,500 test files migrated from Enzyme to React Testing Library, each file passing through just, lint, and type checking in sequence, with errors from whichever layer failed handed to the large model to fix. 75% was migrated in 4 hours, then pushed to 97% over another 4 days—originally estimated at a year and a half of manual work, finished in 6 weeks. Their own conclusion: repeated retries work better than polishing prompts to perfection.

Fuzzing is an automated software testing technique that continuously feeds large amounts of random or semi-random malformed data into a program, observing whether it crashes, hangs, or produces abnormal behavior, thereby discovering defects caused by boundary conditions and unhandled inputs. Its complementarity with unit testing lies in this: unit tests verify scenarios the developer "thought of and wrote down," while fuzzing can cover input combinations nobody anticipated. In practice it's often paired with coverage-guided strategies, letting the mutator prioritize exploring code paths not yet covered. The "continuous fuzzing" mentioned in the article refers to long-running fuzz tasks in continuous integration, suited to filling in security and robustness blind spots that traditional tests can't reach on large-scale AI-generated code.

What If Legacy Code Has No Tests

The hardest case is legacy code with no tests at all. Michael Feathers's definition of legacy code in Working Effectively with Legacy Code is precisely "code without tests." His method: first find the point to change, then find the point that can be tested, break dependencies, write tests, and only then change the code. The tests written at this stage are called characterization tests—they don't judge whether the code is correct; they record current behavior as-is, locking in bugs along with everything else.

This step can now be handed to AI. The exact words from Anthropic's summary: if you can't inherit a judge, have Claude build one—the original codebase is always the standard answer. When Mike Krieger migrated 165,000 lines of Python to TypeScript, he had no ready-made tests, so he had Claude write a script that ran 7 real scenarios on both old and new sides and compared outputs, fixing until all 7 passed: with the judge in place, he also deliberately broke the code to confirm it would actually turn red.

Characterization Test is a concept proposed by Michael Feathers in Working Effectively with Legacy Code, referring to tests written before modifying legacy code to lock in the code's current actual behavior. Unlike regular tests, it doesn't use "correctness" as its standard—even if the code has bugs, the test asserts the bug's behavior as-is, with the goal of establishing a "behavior snapshot" so that any unintended change to existing behavior during subsequent refactoring or migration is immediately caught. The typical process for writing characterization tests: find observable output points in the code (return values, logs, side effects), write assertions for them, then run and record actual results as expected values. Because it only cares about "whether it changed" rather than "whether it's correct," it's very well suited as a judge during AI migration: the old system is the standard answer, and the new implementation must align byte-for-byte with old behavior.

Tests Are AI's Exit Condition, Not Proof of Quality

Back to Burn's 19 regressions. Andrew Kelley, creator of the Zig language, publicly questioned whether this test suite was enough to cover 1 million lines of code nobody had reviewed line by line. Burn later conducted 11 rounds of security audits, attached continuous fuzzing to every parser, and fixed another 15 PRs. This shows that even with all tests green, "passing tests" cannot be equated with "meeting quality standards."

So before handing a legacy codebase to AI, ask three things: Is there a command that can tell it whether the change is correct? Is that command currently green on the main branch? If you deliberately break something, will it turn red? If all three are yes, then let it draw the map itself—you just need to fill in the few lines it can't read from the code.

Key Takeaways:

• Tests are the exit condition for AI code changes, determining when it stops and whether it changed things correctly
• The map should be drawn by AI; humans only need to fill in the implicit project requirements it can't read
• Checks run from cheap to expensive; any stage failing means the change is discarded outright
• When legacy code has no tests, have AI generate characterization tests to lock in existing behavior, bugs included
• All green doesn't mean quality standards are met—fuzzing and security audits still need to fill the gaps

Background Notes

Semantic Regression refers to a subtle but real change in behavior at the semantic level even when code passes syntax, compilation, and all existing tests. Such issues are hard for tests to catch because tests only assert the portion of the behavior contract the author explicitly wrote down; untested semantics—such as floating-point precision handling, null boundaries, error code mapping, iteration order—once quietly changed during a rewrite, can cause hard-to-diagnose failures downstream. This is also the technical root of "all green doesn't mean quality standards are met": test coverage is essentially "the proportion of behavior with written assertions," not "proof of correctness for all behavior." Therefore, after large-scale automated migration, differential testing, fuzzing, and human review are usually needed to cover semantic-level gaps.

One-line Summary

Before handing a legacy repository to AI, give it tests as a judge first, then let AI draw the map itself.

Paragraph Summary

The article proposes a counterintuitive migration order: before handing a legacy codebase to AI for refactoring, provide reliable tests rather than a detailed AGENTS.md. The reason is that tools like Claude Code stop when they "think the job is done," and tests are their only exit condition and correctness judge. The map (file dependencies) can be drawn by AI and deterministic tools themselves; humans only need to supplement project-specific implicit requirements. Checks should run from cheap to expensive, and any stage failing means discarding the change. For legacy code without tests, have AI generate characterization tests to lock in existing behavior. But all-green tests don't mean quality standards are met—Burn still saw 19 semantic regressions after migration, requiring fuzzing and security audits to fill the gaps.