Zero Data, Full Pipeline: How a Test Agent Auto-Generates Executable Test Cases

Title

Zero Data, Full Pipeline: How a Test Agent Auto-Generates Executable Test Cases

Meta Description

A test agent demo shows the complete flow from an empty system to a test report: automatically identifying API parameter constraints, generating executable cases with multi-layer assertions, and batch review exposing config errors—all without any manual case edits before running.

Article Content

A developer spent over four months building a test agent and recorded a screen demo showing the entire process from an empty system to a test report. The most notable thing about this video isn't the feature list—it's that it answers a widespread industry doubt: can AI-generated test cases actually run directly?

Starting from Zero: Project, Environment, and API Import

The demo starts from a completely wiped system—all data tables cleared, no projects, no environments, no APIs. The first step is creating a project and selecting the API automation type, then creating an environment. For privacy reasons, the environment details are hidden, but the process itself skips no steps.

API import supports two methods, and the demo chooses to read and parse a local request payload. After uploading a payload, the system parses out two APIs. The author selects the homepage API and sends a request directly; the status code returns 200 and the response data looks normal, proving the project and environment are configured correctly.

This step seems basic, but it's an easily overlooked validation point: many AI testing tools don't guarantee environment connectivity after API import, whereas here "sending a real request" is treated as a check before moving into case design.

Case Design: Parameter Constraint Identification Is Key

After selecting the API and clicking "Design Test Cases," the system begins a series of operations: locking the API, querying the corresponding data tables, and extracting request parameters. On the first run, it automatically identifies the Chinese meaning, data type, and value range of each parameter, and asks for one manual confirmation.

The demo shows specific identification results: an enum field is identified as having only two values, 1 and 2; a status field is constrained to the range 1 to 12; password length is limited to 6–8 characters; and the phone number field is marked as needing to match a real number format.

This constraint identification directly determines the quality of the generated data. The author specifically emphasizes that the phone numbers generated by the system are real numbers conforming to the specification, not random digits; values for fields like username also fall strictly within the preset ranges rather than being filled arbitrarily. The cases cover testing methods such as orthogonal combinations, boundary values, missing options, and missing required fields, and each case is tagged with a priority.

Orthogonal combination is a systematic test case design method whose core idea is to cover all pairwise interactions among multiple input parameters with the fewest cases, rather than exhaustively enumerating the full Cartesian product. For example, if three parameters have 3, 4, and 5 possible values respectively, the full combination requires 60 cases, while an orthogonal array can compress this to a dozen or so cases while still guaranteeing that every value pair of any two parameters appears at least once. In the context of AI-generated test data, orthogonal combinations can significantly reduce the number of cases and control execution costs, while avoiding situations where certain parameter combinations remain uncovered for long periods due to random sampling. It is usually used together with boundary value analysis: boundary values verify the two ends and critical points of a value range, while orthogonal combinations verify the interactions between parameters. Only by combining the two can a balance be struck between case count and coverage.

Assertions and Database Validation: Beyond Request Parameters

More noteworthy is the assertion layer. Each generated case contains five layers of assertions: the first layer validates the outermost status code, and there are also assertions on the response body and on the database.

The API in the demo includes both create and edit operations, and the system generates corresponding assertion logic for each scenario. For business constraints like username uniqueness, the system automatically generates a delete operation—because if this create uses a certain username, using it again next time would trigger a duplicate prompt; the delete action ensures data independence between cases.

In other words, the generated cases require no manual modification of request parameters, priorities, specific values, or assertions—they are executable out of the box.

Data independence between cases is a prerequisite for whether automated tests can be rerun stably. If multiple cases share the same username, order number, or unique key, then after the first case writes data, subsequent cases using the same value will trigger uniqueness conflicts, causing cases that should pass to fail—this type of failure is called "case contamination." It is unrelated to product defects but can seriously interfere with interpreting test results. There are three common solutions: first, as described in the article, automatically generate delete or rollback operations within cases to clean up data after execution; second, generate unique values with timestamps or random suffixes for each execution; third, use test fixtures to uniformly prepare and destroy data before and after cases. For APIs that involve database writes, the assertion layer also needs to validate the persisted results; otherwise, checking only the HTTP response body may miss the problem of "the API returns success but the data is not correctly persisted."

Batch Review: Exposing Problems Before They Enter the Library

After generation comes the review stage. The 32 cases are executed in batches of 10 per group, with execution status shown in real time on the left, and a summary result at the end.

The review results fall into three categories: 9 cases that pass directly and can enter the library, cases with defects, and cases requiring manual confirmation. In the demo, one failed case appears because the parameter constraint model was mistakenly set to "user already exists," causing the generated request parameters to differ from expectations—which precisely illustrates the value of the review mechanism: it exposes model configuration errors before they enter the library, rather than waiting until the test execution stage to discover them.

The author points out that such problems usually stem from mistakes when first designing the constraint model, and after correction, regeneration is all that's needed.

Review gate refers to a manual or automated review step set up before test assets formally enter the case library, with the goal of intercepting low-quality cases upstream. Once AI-generated cases are directly added to the library and enter the regression suite, erroneous cases will continuously produce false positives, and the team will gradually lose trust in the results, eventually abandoning the entire suite. Batch review usually focuses on three types of issues: whether the configuration the generation depends on (such as the parameter constraint model) is correct, whether the expected values of cases are consistent with business rules, and whether there are data dependencies or execution-order coupling between cases. Moving review forward to before library entry makes the cost of fixing far lower than troubleshooting false positives during test execution, and also makes it easier to feed discovered issues back into correcting the constraint model, so that subsequent generation quality continues to improve.

Execution Verification: Unmodified Cases Run Directly

After passing review, the 9 cases enter the library directly, and the author emphasizes that no manual modification was made to any case throughout the entire process. Then the tests are run directly.

The execution results show expected values and actual values case by case, and failed cases include the actual returned results as evidence. From creating the project, configuring the environment, parsing APIs, designing cases, and reviewing to executing tests, the entire process is completed, and no manual editing was done during the case design stage.

The author finally compares it with similar tools on the market: previous tools often still required manual editing of request parameters, priorities, and assertions after generating cases, while this system's difference lies in automating these steps as much as possible. The constraint model only needs to be confirmed once during the initial design, and subsequent case generation for similar APIs reuses it directly without repeated pop-ups.

What This Demo Shows

A common criticism of AI-generated test cases is that "generation is easy, execution is hard"—non-compliant parameters, missing assertions, and cross-contamination between cases can all turn generated cases into dead weight. This demo targets exactly that pain point: constraint-driven data generation ensures parameter validity, multi-layer assertions cover both responses and the database, and delete operations solve the case independence problem.

Of course, a single-API demo is still not enough to prove performance under complex business flows, and the accuracy of the constraint model also depends on the quality of the first manual confirmation. But at least in this workflow, the chain from zero data to executable cases is closed.

One-line Summary

A developer demonstrates an AI test agent generating directly executable API cases from scratch

Paragraph Summary

A developer spent over four months building a test agent and used a screen recording to show the complete chain from wiping the system to producing a test report: creating a project and environment, importing APIs and sending a real request to verify connectivity, automatically identifying parameter types and value ranges with one manual confirmation, generating cases covering orthogonal combinations and boundary values, and after batch review, 9 cases entering the library directly and executing successfully. The core value of the demo is responding to the doubt of "can AI-generated cases run directly": constraint-driven data generation ensures valid parameters, multi-layer assertions cover status codes, response bodies, and the database, uniqueness constraints automatically generate cleanup operations to ensure case independence, and no manual modification is made throughout. The author also acknowledges that a single-API demo is not yet enough to prove performance under complex business flows.