A Complete Method for AI Agent Test Case Execution: From Planning to Evidence Chain Closure

Introduction: Writing Test Text ≠ Completing a Test

In a hands-on demo video, Bilibili creator 奇遇AI made a point that cuts right to the heart of AI testing: Just because an AI can write test text doesn't mean it has actually completed the test.

What truly matters isn't seeing a green "Pass" appear on screen at the end — it's that every conclusion can be traced back to a specific request, a database change, and a cleanup record. In other words: conclusions must be grounded in evidence.

This methodology isn't theoretical. The author built it from scratch in a real company project, refined it through repeated iteration, and then redesigned it as a standalone public demo — a "voice chat room" app — walking through the entire process from preparation and planning to execution and verification. This article traces that complete chain, examining how an AI Agent shifts from "improvising freely" to "executing against a checklist and letting evidence do the talking."

Setting Up the Working Directory: Anchoring the AI Agent's Context

When most people try to replicate AI-driven testing, their first instinct is to craft a carefully worded prompt. But the author emphasizes that the first thing to prepare isn't a prompt — it's a clear working directory.

This directory needs to contain everything in a single context: requirements documents, API specs, backend code, table schemas, test accounts, seed data, and rules for saving results. The directory defines the boundary of this execution run — scripts are the entry point, and results flow back into this same directory. When all the reference material lives in one context, subsequent actions can more naturally "emerge" from it.

A key safety design: test accounts are granted read-only permissions. The database Observer can only read data through pre-defined read-only queries, preventing any unintended changes during execution. This allows result verification without polluting the environment.

The author also offers two practical usage modes: for short-term trials, simply prepare a zip file and unzip it for Codex to open; for long-term iteration, a fixed working directory is preferred so context can accumulate over time.

Breaking Down 18 Verifiable Cases: Covering Five Risk Categories

The execution scope is explicitly locked to 18 Cases, rather than letting the AI Agent improvise. These 18 cases cover five major risk categories:

• Normal and boundary conditions: Verify that legitimate redemption flows succeed
• Invalid input: Verify that bad inputs are correctly rejected
• Authentication and authorization: Verify that identity boundaries are enforced
• Balance validation: Verify that balance checks are in effect
• Replay (idempotency): Verify that duplicate submissions don't cause duplicate deductions
• Data isolation: Verify that data from different users doesn't bleed across accounts

Each case maps to a specific risk. The Runner executes strictly against this list — no cases are added on the fly, and expected results are never modified from memory.

The value of this grouping is that each failure type has a corresponding focus area: normal scenarios check business flow closure, abnormal scenarios check for side effects, permission scenarios check identity boundaries, replay scenarios check idempotency, and isolation scenarios check for data leakage. Fix the questions first, and the results will have clear boundaries.

The author also notes that files aren't just stacked in parallel — each has a distinct responsibility: the API doc defines the external contract, routing receives incoming requests, the parameter model rejects invalid input, the Service layer drives business logic changes, SQL defines which tables data lands in, and the Observer handles controlled reads only. With responsibilities separated, any inconsistency can be traced to either the input layer or the data layer.

Treating Each Case as a Documented Experiment

The author introduces a sharp analogy: a test case is like a documented experiment. The complete execution chain is fixed to five steps:

1. Save db_before (database state before the request)
2. Send the real request
3. Save db_after (database state after the request)
4. Run assertions and write evidence
5. Execute cleanup

This places the API response, database changes, and cleanup results on the same chain. Every case answers the same set of questions: What was sent? What did the service return? What changed in the database before and after? Did cleanup return things to the starting point? Looking at only one layer leaves the conclusion incomplete.

The pre-execution baseline is also explicitly fixed: contribution value 1650, points 1000, orders and both types of ledger entries all at 0. Every case starts from this reproducible baseline.

Dry Run and Approval: The AI Agent Cannot Start on Its Own

Before formal execution, a Dry Run is required. The Dry Run only inspects the plan — it does not call any real business APIs. Its purpose is to surface the execution scope and potential risks before anything runs.

This reflects an important control principle: the AI cannot start on its own. The author first reviews the scope, permissions, and cleanup requirements, and only after confirming everything is correct does they generate a time-bound Approval that locks down what is permitted in this run. Only then does the Runner proceed into the 18 cases.

This design keeps "approved scope" and "actual actions" as separately verified items. Before the Runner starts, no "Pass" results appear on the page prematurely. The plan file cannot substitute for execution evidence — this is the iron rule throughout.

Validating the Evidence Chain with Three Representative Cases

After execution, the overall results are reviewed first: total count, pass/fail breakdown, number of evidence entries, and cleanup status. But the summary only shows whether the run completed — it cannot replace specific evidence. The author walks through three representative cases individually.

Normal Redemption: Three Layers of Evidence, All Required

Input conditions, API response, and pre/post database states must all correspond to each other. Even if the API returns 200, without a corresponding database change, you cannot prove the business transaction fully succeeded. How much the contribution value decreased, how much the points increased, how many orders and ledger entries were created — all of these should be derivable from business rules and must match the actual query results line by line. Any discrepancy means it cannot be classified as a success.

Invalid Input: Rejection Must Happen Before Any Write

The author deliberately passes in a decimal value of 10.5 as a test. The same principle applies: first confirm what the service rejected, then confirm the rejection happened before any business write, and finally use the pre/post database states to prove no "half-record" was left behind.

The error message and the absence of side effects must both hold true for a failure to be considered "clean." The database state is completely unchanged before and after — no balance deducted, no points added, no order created, no ledger entry written.

Replay Test: Validating Idempotency

The same requestId is submitted twice in succession. The first submission should have replay set to false; the second should be recognized as a duplicate with replay set to true, and both responses should return the same order number (the order number is not hardcoded in advance). The core of replay validation: the same requestId must produce only one business change. The second request may return an "already processed" marker, but must never trigger another deduction — and the database change must appear only once.

Cleanup Verify: The Final Gate

The test run isn't done when execution ends. Cleanup Verify must confirm that contribution values and points are back to baseline, and that orders and both ledger types are zeroed out. The key point is that cleanup isn't about changing numbers on a page — it's about re-reading the real baseline to confirm that accounts, orders, and ledger entries have all returned to a reusable state.

The author puts it plainly: being able to execute is only the first step; being able to restore is what gives the next test run a trustworthy starting point. Whether the next round of tests is valid depends entirely on whether this step is clean.

Conclusion: A Traceable Evidence Chain Is the Core Value of AI Testing

This complete closed loop — from material preparation to final environment restoration — was fully executed. All machine records preserve the facts; summaries only serve as quick navigational aids. If you ever need to trace where a conclusion came from, you follow the index back to the specific file, not point to a one-line summary on the page.

The author notes that the project's testing skills have been open-sourced and are free to use — both TMs and QAs can reuse and build on them, with continuous updates planned. Future additions include more complex group chat flow testing, and a no-documentation JS reverse-engineering skill — using traffic capture and frontend JS to reconstruct real APIs, parameters, and call sequences, ultimately letting AI complete the API orchestration.

For teams exploring AI Agent-based testing, this methodology of "conclusions must be grounded in evidence" is arguably worth more than any flashy automation demo.