AI as QA Tester: A 40-Bug Experiment Reveals Surprising Truths About Prompt Design

Title

AI as QA Tester: A 40-Bug Experiment Reveals Surprising Truths About Prompt Design

Meta Description

An experiment with a virtual e-commerce site hiding 40 bugs used three prompt levels to have Claude Code act as a QA tester. Result: a full checklist raised bug detection from 62% to 76%, but a more detailed role prompt performed worse. A deep dive into the limits of AI agents as QA testers and what it means for prompt design.

Article Content

Automated tests only catch the problems you've already written assertions for. A button works but nobody can find it, a form submits but the phone number format takes repeated guessing — these experience problems that pass CI yet confuse users are where QA's real value lies.

A natural question follows: can AI agents handle this part of the work? Not detecting whether a page loads, but judging "would a real user get confused here?" And further, how much does this ability depend on how you ask?

A developer built a virtual coffee e-commerce site hiding 40 bugs, then used three prompts of varying detail to have Claude Code act as a QA tester. The results upended quite a few intuitions.

Experiment Design: A Coffee Shop Site and 40 Known Answers

To test a tester, you must know the correct answers in advance. A real website can't provide that condition — if the agent reports 5 issues, you have no way to tell whether it missed 10 or there were only 5 in total.

So the experimenter built Ponto Coffee Roasters: a virtual online store with a shop page, product page, basket, two-step checkout, and registration form. No real server, just a small piece of code simulating backend behavior, with random delays and a setting where the first submission always fails — how an app behaves while waiting and when errors occur is itself an important part of user experience.

The 40 bugs came from two different AI models. Claude built the store and planted the first 20 bugs; then the experimenter asked Codex (another AI) to introduce 20 more in a separate request Claude couldn't see. This was done to avoid having both the bug-planting and bug-finding come from the same model.

The bugs covered a wide range: nearly unreadable light-gray text, an "Add to Bag" button that gives no feedback when clicked, a review step showing "free shipping" but quietly including €6.90 in the total. The basket's minus button actually increases the coffee quantity, the logo link points to the basket, and the mobile checkout page overflows the screen. Some were subtler still: the search box clears itself when you click elsewhere, and the last name field stops accepting input after 8 characters.

Three Ways of Asking: From a One-Line Instruction to a Full Checklist

The experiment changed only one variable: the prompt.

Level 1 is the most common one-line instruction: "You are a QA engineer. Test the website and find UX errors."

Level 2 gives the agent a role (a senior QA tester focused on user experience), provides store background, a working method (walk the main flow first, then test error scenarios and small screens), and a report format.

Level 3 adds a full checklist on top of Level 2, based on Jakob Nielsen's 10 usability heuristics and WCAG accessibility guideline basics. It also asks the agent to test in rounds: desktop, error scenarios, waiting and failures, mobile dimensions, keyboard-only operation.

The checklist was written before any bugs existed, avoiding a checklist that describes the answers rather than the testing dimensions.

Jakob Nielsen's 10 usability heuristics were proposed in the 1990s by Jakob Nielsen, co-founder of the Nielsen Norman Group, and are among the most cited usability evaluation frameworks in human-computer interaction. They include ten dimensions: visibility of system status, match between the system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, aesthetic and minimalist design, help users recognize and diagnose errors, and help and documentation. These principles don't target specific interfaces but offer a general set of checking perspectives, often used in heuristic evaluation — having a few evaluators walk through an interface against the principles and list violations.

WCAG (Web Content Accessibility Guidelines) are published by the Web Accessibility Initiative under the W3C and are internationally recognized accessibility standards. They are organized around four core principles: perceivable, operable, understandable, and robust. Each principle has several success criteria, divided into three conformance levels: A, AA, and AAA. Unlike Nielsen's heuristics, which focus on general usability, WCAG focuses more on whether users with disabilities — including visual, auditory, motor, and cognitive impairments — can perceive and operate content, such as keyboard accessibility, color contrast, focus visibility, and screen reader compatibility. Combining both into a checklist covers both ordinary users' experience pain points and the accessibility compliance baseline.

Fairness Controls and Scoring Method

Each run used the same agent and model: Claude Code (Anthropic's coding agent) running Claude Sonnet 5.5, driving a fresh browser instance via Playwright. The browser was its only tool — no file reading or terminal access, so it couldn't view source code or the answers. It could only use the site like a visitor. Each level was run three times, because AI output has randomness.

Each report was scored against the answer key: clearly describing a bug earns full credit, merely mentioning it earns half, and missing it earns nothing. This is called a weighted detection score. To reduce scoring subjectivity, an independent AI agent scored all nine reports without knowing the experimenter's scores, and the two never differed by more than half a point per report.

Results: The "Better" Prompt Performed Worse

Prompt	Three-run scores	Average	Percentage	Time per run	Cost per run
Level 1: one-line instruction	24, 25, 25	24.7 / 40	62%	1.7 min	$0.51
Level 2: role + background	21, 22, 23.5	22.2 / 40	55%	1.9 min	$0.51
Level 3: full checklist	27.5, 31.5, 32	30.3 / 40	76%	3.5 min	$0.81

The first surprise is in the middle row. The experimenter expected each level to beat the previous one, but Level 2 lost to the one-line instruction.

Level 2 produced longer, more polished entries, each with a title, location, impact, and severity — yet scored lower. The one-line runs wrote fast, dense lists. The report format may have contributed, but these runs couldn't isolate the format's effect from differences in how the agent explored. This is a pattern worth testing, not a conclusion.

The Checklist Changed Not Just Wording, but Behavior

Level 3 didn't just describe better — it did different things.

It was the only level that pressed the Tab key — a way to test how users without a mouse experience the site, and the reason all three Level 3 runs found "no visible keyboard focus." None of the other six runs caught it.

Every run, at every level, shrank the window to mobile width at some point. But only Level 3, in two of three runs, found that the checkout page overflows on mobile. Neither Level 1 nor Level 2 runs reported it, and one even declared "no horizontal scroll at 375px."

The checklist acted like a test plan. It didn't make the agent smarter; it sent it to places it would otherwise skip.

Checklists Also Narrow Attention

The opposite happens too. The footer's newsletter box had a spinner that never stops. One one-line run clearly reported it, and two others got partial credit. The six runs with detailed instructions scored nothing — they followed the given flow paths, and the footer wasn't among them.

This phenomenon is just as common with human testers. Give a tester a checklist and they'll cover what's on it. Free exploration finds strange problems; structured methods find thorough ones.

The distinction between test plans and test charters helps explain this. A test plan is usually a predefined, structured document listing the features, scenarios, and checks to cover, emphasizing repeatability and coverage; a test charter comes from exploratory testing methodology, where a tester freely explores within a time box and records findings, emphasizing discovery of unknown problems. The two aren't opposed: structured methods ensure known risks are systematically covered, while exploratory methods are used to find problems the plan didn't anticipate. Giving a tester a checklist is like handing them a test plan — they tend to tick items off, allocating attention to what's on the checklist and reducing incidental discovery outside it. This is called checklist-driven attention narrowing, and it's why many teams keep exploratory testing sessions beyond regression testing.

Bugs All Nine Runs Missed

Five bugs were missed by all nine runs. Four came from Codex's set: the search box accepts only 12 characters, the search box clears when you click elsewhere, the last name field stops at 8 characters, and the Huila Sunrise coffee card on the shop page shows an image of a different coffee.

The fifth came from Claude's set: double-clicking "Add to Bag" adds the item twice.

Most of these share one thing in common: they produce no obvious error message or broken layout. Finding them requires trying boundary values or comparing behavior across interactions. No report identified the image mismatch between the shop page and the product page.

The double-click bug is a more complex case. Codex's set also established that a single click adds two items, so a double-click adds four. Since a single click already adds two, the duplicate-click problem is harder to distinguish. Planted bugs can mask each other.

Boundary value analysis is a classic black-box technique in software testing. It rests on the observation that errors tend to cluster near the boundaries of an input domain rather than at middle values. So testers focus on the minimum, maximum, just-past-the-boundary, and near-boundary values — for example, when a field allows 12 characters, test 11, 12, 13, and longer strings. When the search box truncates at exactly 12 characters or the last name field stops accepting input after 8, only actively entering overly long or extremely short values can trigger it. Such defects throw no error prompt and break no layout, making them hard for an agent to notice by simply walking the normal flow.

Complementing it is cross-interaction comparison, comparing the behavior of the same information or control across pages or states. For example, if the shop page and product page use different images for the same coffee, you won't find it without actively comparing; a button that adds two items on one click and four on a double-click also requires comparing single- and double-click results to identify. Both methods require the tester to actively construct atypical operations rather than relying on the interface's own feedback signals.

Bugs Everyone Found

On the other hand, every run reported the logo pointing to the basket and the password displayed in plaintext. Equally consistent findings included: shipping labeled "free" that actually charges, basket rows that ignore quantity, and price changes between pages — all nine runs caught these numerical inconsistencies, including the one-line runs.

Scorers flagged at most one false positive per run, mainly about slow loading caused by the test environment rather than the app itself. Every run also found real problems nobody planted, such as form fields with no autocomplete suggestions, error messages screen readers wouldn't announce, and the header still showing "Create Account" after registration.

Limitations of the Experiment

This is a small-scale experiment. Nine runs on a single model is a hint rather than proof, and a different model could change the numbers. Testing ran in a cloud sandbox that blocked web fonts, slightly differing from a real visitor's experience. All 40 bugs came from AI (two different models), and human testers might plant different types of problems. Some planted bugs interacted: a cropped product description masked a wall-of-text problem, and the duplicate "Add to Bag" obscured the double-click bug. This changed the visibility of some bugs and means the five universally missed ones don't all point to a single weakness in the agent.

Next Step: A Checklist That Learns

The checklist made the agent thorough, while the one-line instruction's freedom made it curious. So the suggestion is to run two complementary processes before each release: a checklist run and a free-exploration run — the latter tasked with finding bugs not yet on the checklist.

Looking back at the saved scores, the experimenter paired runs in twos and gave each bug the best score from either run. One free run plus one checklist run averaged 31.6/40, while two checklist runs averaged 32.3/40. This is a retrospectively calculated workflow rather than an independent test, but it suggests the free run doesn't raise the score — its value lies in finding problems not yet on the checklist.

Those findings, along with human testers' findings and real users' complaints, feed back into the checklist. Some become general habits (like "enter unusually long values in every field"), kept because they require judgment. Others are specific regression issues (like "the newsletter spinner must stop"), for which the checklist is a cheap placeholder — adding a line takes seconds, while writing an end-to-end test with a tool like Cypress takes real investment. So ship the checklist first, and graduate it to Cypress tests when there's time or the checklist grows long.

Can AI Be Your QA Tester?

In this small experiment, the browser-driven AI quickly and cheaply found many planted functional and usability problems. The best configuration reached a 76% detection rate at under a dollar per run. The checklist improved coverage, but several subtle bugs were missed in every run. It's suited as a first round of screening before release, not as a replacement for the human tester who types a long name, accidentally clicks the wrong spot, and then thinks "wait, didn't this look different a moment ago?"

The core takeaway about prompts is this: an AI tester's prompt works much like a test plan. What you put in gets carefully covered, and what you omit may never be visited. And like any good test plan, it should never be finished — every bug the agent misses today is a candidate line for tomorrow's checklist.

The whole experiment is open source: the store, the 40 bugs, the three prompts, and every agent's report and score. You can run your own prompt on GitHub and see how many you can catch.

Have you tried using an AI agent for QA or UX review? Feel free to share what it found for you, and what it walked right past.

One-line Summary

An experiment with AI as a QA tester shows a full checklist raised bug detection from 62% to 76% but narrowed exploration.

Paragraph Summary

A developer built a virtual coffee e-commerce site hiding 40 bugs and used three prompt levels to have Claude Code act as a QA tester, running each three times with weighted scoring. Results: the full checklist raised detection from 62% with the one-line instruction to 76%, at a cost of $0.81 per run versus $0.51; but Level 2 with a role and background dropped to 55%, worse than the one-line instruction. The checklist changed the agent's behavior, not just its wording — only runs using it pressed Tab and found the missing keyboard focus. Five bugs were missed by all nine runs, sharing the trait that they produce no obvious error or broken layout and require boundary testing or cross-interaction comparison. The author suggests using checklist runs and free-exploration runs complementarily and feeding findings back into the checklist.