AI Testing in a Closed Loop: Full Pipeline from API Collection and Test Case Generation to Agent-Driven Execution

In software testing, the promise of AI adoption has been talked about for years — yet closed-loop implementations that actually work in enterprise environments remain rare. This article, based on a technical talk, walks through a complete AI-powered testing workflow covering API collection, documentation enrichment, test case generation, Agent-driven automated execution, and test report generation. Compared to the superficial "upload your docs and get instant test cases" demos floating around, this approach addresses the real pain points of enterprise adoption.

Two Real Challenges in AI-Driven Testing

Most people imagine AI testing as: "throw your API docs at the AI, and it spits out perfect test cases." In practice, two hard problems surface immediately.

Challenge 1: Incomplete information. Real-world API documentation is almost always incomplete. The most common gap? The vast majority of API docs have no response body examples. Without a response schema, any assertions AI generates are essentially useless fluff. As the presenter put it: "If I give you an API and you know how to send the request but have no idea what the response looks like, you can't write meaningful assertions."

Challenge 2: AI isn't as smart as you think. AI output must go through human review. This was the most emphasized point in the entire talk: "Just because AI generated a bunch of documents doesn't mean you can use them directly." If the APIs captured by the AI aren't actually accessible, every test case downstream is worthless.

These two challenges define the core design principles of the entire workflow: information must be actively enriched, and outputs must be reviewable and editable.

Step 1: API Collection and Documentation Enrichment

Rather than asking AI to "make something up" in the face of missing documentation, the solution is to equip it with an API collection tool. By attaching a recording floating button to the browser, it's possible to capture in real time the API requests triggered by page interactions, then feed the captured traffic to AI for analysis.

This browser recording widget is essentially a lightweight network traffic interception tool — conceptually similar to Charles or Fiddler — that intercepts XHR/Fetch requests via injected scripts or a proxy layer, capturing request headers, request bodies, and response bodies in full. What sets it apart from traditional packet capture tools is that the "record" action is embedded directly in the testing platform, so captured results can be stored in structured form and piped straight to AI for analysis, eliminating the need to manually parse HAR files. The entire chain — capture → structure → AI analysis — becomes seamless and automatic.

This solves the "information source" problem: the AI can analyze not just individual APIs, but the full chain of API calls triggered by user interactions on screen. After capture, the AI automatically organizes endpoints — homepage APIs, trending recommendations, click rankings, and more — into structured documentation covering endpoint IDs, names, URLs, parameters, and other metadata.

The critical piece is the validation step: every captured endpoint must be verified to confirm it's reachable, parameters are correct, and descriptions are accurate. Even testers who have never touched the project before can complete API registration and naming with the collection tool. This validated API documentation becomes the foundational data dependency for all subsequent AI test case generation.

Step 2: AI-Generated Test Cases and Test Points

With complete API documentation in place, AI can generate API test cases from the docs, or generate targeted cases for specific modules on demand.

For Web UI testing, the process is similar: ask the AI to analyze a page, automatically extract test points in mind-map form, then generate editable, searchable test cases — including test steps and expected results — rather than a static document.

Worth highlighting is the multi-terminal testing capability. AI can directly read and operate PC applications (e.g., the Windows Calculator), Web browsers, Android devices, and even HarmonyOS devices. The underlying tech stack is straightforward:

• Android: ADB
• Web: Playwright
• PC (desktop): PyAutoGUI
• Visual recognition: Vision LLMs

Each tool has a clear role, together forming an automation infrastructure that covers mainstream platforms. ADB (Android Debug Bridge) is Google's official command-line tool supporting app installation, shell command execution, screenshots, and more — the backbone of Android test automation. Playwright is Microsoft's modern open-source Web automation framework, offering more stable waiting mechanisms and better async support than Selenium. PyAutoGUI enables cross-platform desktop interaction by simulating mouse movement and keyboard input. Vision LLMs (such as GPT-4V or Claude Vision) give AI the ability to "see and understand" interfaces without requiring pre-annotated UI elements, making them an important complement to traditional control-tree-based approaches. In practice, this means AI can "see the interface, understand it, and plan test cases" — for projects it has never encountered before, it can independently identify what's on screen, map out modules, and execute operations, which is exactly what automated testing was designed to do.

Step 3: Bridging the Gap Between Manual and Automated Testing

This is the most easily overlooked — and most critical — part of the entire pipeline. AI can operate UI and generate manual test cases, but that doesn't mean it can directly run automated tests.

There is a clear "gap" between manual test cases and automated ones. Take a login scenario: where do the username and password come from? In multi-environment setups (production, UAT, test), databases and accounts differ across environments. If test data is hardcoded, AI can't flexibly switch between them. Real automation also involves:

• Dynamic data retrieval and cleanup (e.g., querying databases)
• Dynamic function execution
• Data chaining (output from API A passed as input to API B)
• Encryption and decryption handling

None of these can be solved by simply telling AI to "just run it." The solution is to have AI perform automated orchestration, not blind execution. AI arranges the API content into normalized step-by-step flows, and testers review and edit each step.

AI can automatically generate assertion logic — checking whether an API returns JSON, whether the JSON contains specified fields, and so on. It can also generate code to dynamically read login credentials from a database and run it directly within the automation flow. The entire pipeline — from data retrieval, login, and assertion all the way to the test report — is fully visible and transparent.

Engineering for Production: Skills, Knowledge Bases, and Harness

If the sections above describe the "process," what actually elevates AI testing from a toy to a productive tool is the establishment of an engineering infrastructure.

Skills: Standardized Encapsulation of Testing Capabilities

Capabilities like API collection, test point analysis, and desktop automation are all essentially Skills. In mainstream Agent frameworks (such as LangChain, AutoGen, or Claude's Tool Use), a Skill corresponds to the concept of a "Tool" — the LLM uses a Function Calling or Tool Use mechanism to "decide" during inference which tool to invoke and what parameters to pass; the external code executes and returns the result to the model to continue reasoning. Packaging testing capabilities as standardized Skills is fundamentally about building a "toolbox for the testing domain": each Skill has a well-defined input/output specification, the model can call it like an API, and implementation details remain transparent to the model, making Skills easy to reuse and migrate across platforms.

Skills are not tied to any particular platform — they can be used across self-built agents, Claude Code, and other Agent environments. The core enterprise adoption strategy is: encapsulate the capabilities of every project and every tester into Skills, standardize the specs, and ensure that everyone produces test cases of consistent quality.

Knowledge Bases: Filling AI's Blind Spots on Business Context

For complex business processes with intricate API call sequences, AI can handle simple projects well enough — but for complex ones, it will almost certainly be lost. This is where knowledge bases come in: feed the AI your company's API call conventions, project-specific terminology, and design specifications so it can map out workflows according to correct business logic, rather than guessing.

On the technical implementation side, a knowledge base typically corresponds to a RAG (Retrieval-Augmented Generation) architecture: internal enterprise documents are vectorized and stored in a vector database; when AI executes a task, the system first retrieves the most relevant document fragments for that task, then injects them into the Prompt context. This compensates for the fundamental limitation of LLMs having no access to private business knowledge from their training data. Compared to direct fine-tuning, RAG has extremely low update costs and is well-suited for enterprise environments where business documentation changes frequently — making it the most widely adopted approach for "injecting private knowledge" into enterprise AI deployments.

Harness: Keeping Agents Running Sustainably and Under Control

The talk introduced a key concept: Harness. The term originates from software engineering's "Test Harness" — originally referring to the complete testing framework (drivers, stubs, and auxiliary facilities) that surrounds a component under test. In the context of AI Agents, the meaning expands further to mean "the systematic engineering that keeps an Agent running continuously, cyclically, and under control," encompassing constraint mechanisms, feedback loops, workflow control, and continuous improvement. Think of it like a PID controller in industrial automation: it provides driving force (task triggering), introduces constraints (rule enforcement), and closes a feedback loop (execution result evaluation), preventing the Agent from producing unpredictable side effects in an open environment. Simple operation tools can neither constrain AI from doing things outside its scope nor collect execution feedback — an engineering system is what moves AI from toy-grade to production-grade.

On model selection, the pragmatic approach is to match the model to the scenario rather than chasing the most powerful option. Test execution ultimately comes down to page manipulation and data management; even the most powerful model won't automatically understand complex business processes. A model that's "good enough" with controllable costs is often the more practical choice.

Conclusion: Engineering Is the Real Dividing Line in AI Testing

This AI testing closed loop communicates a clear methodology:

1. Understand the workflow before obsessing over tools — set aside tools and buzzwords, and prioritize mastering the core chain: "how do manual designs translate into automation?"
2. AI is a capability that needs to be constrained — apply constraints through Skills, rule files, and knowledge bases; all outputs must be reviewable and editable.
3. Engineering is the real dividing line — testers who only know how to generate cases with Skills will struggle to differentiate themselves. What mid-to-large enterprises actually need is someone who can drive engineering implementation and platform-level adoption.

For testing professionals, the barrier to entry in the AI era is no longer "do you know how to use AI tools?" It's "can you turn AI capabilities into a reusable engineering system?" That may be the most lasting insight this practice offers the industry.

Note: This article is based on a single-source technical talk. The specific platforms and Skill encapsulation approaches mentioned reflect the practices of the presenter's organization. Readers are encouraged to adapt these ideas thoughtfully to their own contexts.

Key Takeaways