A Month of Writing Playwright Tests with Claude Code: AI Writes the Tests, Humans Keep the Results Solid

A Japanese QA director spent a month using Claude Code as a "pair programmer" for a production-grade Playwright E2E test suite, and came away with an honest conclusion: Claude Code did not replace QA judgment—it only replaced QA's typing hands.

This was not a toy project. The author (a former SHIFT employee, hailing from Japan's largest QA company) used AI-generated tests directly in a production QA pipeline. In his blog, he documents in detail what AI is genuinely good at in test automation, where it falls flat, and the workflow that ultimately worked.

According to his summary, there are three core findings: Claude Code can generate about 70% of the test skeleton code from natural-language flow descriptions; but without guardrails, it performs poorly on selector strategy and flakiness-resistant waiting; and the best practice is "AI writes, humans stabilize"—every AI-generated test must run 5 times locally and once in CI before it counts as done.

The Heart of the Setup: A Non-Negotiable Convention File

The author's project structure is not complex: a standard Page Object Model directory, plus three Playwright project configurations (desktop, mobile, foldable). The real secret is not in the toolchain, but in what you tell Claude Code before you ask it for anything.

He placed a CLAUDE.md in the repository root with just three hard rules: selectors must only use getByRole / getByLabel / [name=] attributes, never CSS class chains; waitForTimeout is forbidden—only Playwright auto-waiting and expect() polling; every test must be able to run independently and in parallel, with no shared state.

The power of these three lines is described by the author as a "quality leap." Without them, Claude Code produces code like page.locator('.css-1kxonj9 > div:nth-child(3)') and waitForTimeout(3000)—exactly the raw material of a fragile test suite.

This finding applies to any team that wants to bring AI coding agents into their testing process: AI does not automatically know your engineering conventions; you must tell it explicitly and up front.

Page Object Model (POM) is a design pattern widely used in test automation that encapsulates each page (or page component) as a class, in which the page's element locators and executable action methods are defined. Test scripts interact with the page by calling the Page Object's methods rather than manipulating elements directly. The benefit is that when the page structure changes, you only need to modify the corresponding Page Object without touching all test cases, thereby improving the maintainability and reusability of the test code. The author's project uses a standard POM directory structure, meaning the test code generated by Claude Code also follows this pattern, centrally managing page elements and actions, which helps keep tests clear and stable.

Semantic selectors (such as getByRole, getByLabel, [name=]) are the element location methods recommended by Playwright. They find elements based on the element's semantic role, label text, or accessibility attributes, rather than relying on volatile CSS class names or DOM structure. These selectors align with how users and assistive technologies perceive the page, and are therefore more resistant to UI refactoring. For example, getByRole('button', { name: 'Submit' }) will find the button with the accessible name "Submit," and even if the button's styling or nesting structure changes, the test will still pass as long as the semantics remain the same. Conversely, using CSS class chains (such as .css-1kxonj9 > div:nth-child(3)) is very fragile—class names may be hashed by build tools or change with styling adjustments, causing frequent test failures. The article's emphasis on using only semantic selectors is precisely to avoid AI generating fragile tests.

What AI-Generated Tests Actually Look Like

The author gives a real interaction: he asked to "write tests for the contact form: success, server error, pending state. Never hit the real API." Claude Code produced a Page Object with network stubbing, intercepting **/api/contact with page.route, supporting injected status codes and artificial delays, and returning simulated success or error responses.

What you might not notice is the pending-state test—simulating a 1.5-second delay, filling out the form, submitting, and then asserting that the submit button is disabled and the success heading is visible within the timeout. The author admits that when writing this by hand, he "probably wouldn't have bothered."

And this test happened to catch a real defect category that recurs in manual QA: duplicate submission. The button only becomes disabled after React re-renders; on a slow network, two quick clicks mean two POST requests. On a contact form, that's merely annoying; on a checkout page, it's a double charge.

This reveals a counterintuitive value point: AI doesn't just accelerate the tests you would have written anyway—it makes you willing to write the tests you previously considered not worth the effort, such as tedious but high-value scenarios like pending states and error paths.

Three Places Where AI Falls Flat, and the Patches

The author frankly lists Claude Code's failure modes, and this part is the most useful for practical adoption.

First, it trusts the current DOM too much. The generated selectors match the structure that exists right now, including auto-generated class names. The patch is that convention file, plus manual review of every selector during the stabilization phase.

Second, it doesn't know what's worth testing. Say "write tests for the dashboard," and it gives you 15 shallow assertions about text visibility. The test plan must come from someone who understands the risks: payment flows, authentication boundaries, state transitions. The author's approach is to write the plan himself and let AI write the code.

Third, it can't feel flakiness. An AI test that passes once is not done. His stabilization loop is: run 5 times locally with --repeat-each=5, run once in CI, and review traces for silent retries. About 1 in every 10 generated tests needs rework at this stage.

The common thread across these three failure modes: they are all about judgment, not coding ability. AI can write code, but it doesn't know what's worth writing, what structure is durable, or what pass is untrustworthy.

Test flakiness refers to the phenomenon where a test sometimes passes and sometimes fails without any code changes, usually caused by asynchronous operations, network latency, animations, or resource-loading timing issues. In Playwright, flakiness can manifest as a click being executed before an element appears, or an assertion running before a state update. The waitForTimeout mentioned in the article is a hard wait that forces a fixed pause but cannot adapt to changes in actual loading speed—either waiting too little and causing failure, or waiting too long and slowing down tests. Playwright's auto-waiting mechanism and expect() polling can wait dynamically for conditions to be met, reducing flakiness. The author requires every AI-generated test to run 5 times locally and once in CI, and to review traces, precisely to identify tests that pass by chance but are actually unstable, ensuring their reliability.

The Underrated Trick: Testing Viewports Nobody Tests

The author's configuration includes a project most test suites don't have: foldable-half-open, viewport 717×1172, mobile, touch enabled.

717px lands right between Tailwind's sm and md breakpoints. The author uses a Motorola Razr foldable himself and found that this gap hides a surprising number of layout bugs. The cost of one configuration block buys a defect category that high-end Android device users encounter every day.

This detail echoes the earlier observation about pending states: AI lowers the marginal cost of "edge-case testing," making affordable the tests that were previously skipped for being too trivial or too laborious. Viewport boundaries, error paths, asynchronous states—these are exactly where production incidents cluster.

The Real Numbers After a Month

The author's results are restrained: test-writing time dropped by about 60%, but the planning phase takes just as long—what's saved is the typing time; coverage went up because those tedious tests are no longer skipped; maintenance costs are manageable, but only if the selector convention is in place—he has seen AI-generated suites without conventions rot within weeks.

The honesty of this account lies in the fact that it does not claim "AI replaces QA." In the author's own words: Claude Code did not replace QA judgment; it replaced QA typing. Test planning, risk identification, selector strategy, flakiness determination—these are still done by humans. What AI takes over is that 70% of repetitive labor.

For teams evaluating whether to bring AI coding agents into their testing process, this case offers a replicable starting point: write the convention file first, then let AI write tests, and finally stabilize through multiple runs. The tool is not the key—the constraints are.