AI + Automated Testing: Building an Engine-Agnostic Quality System with Skill + CLI

Introduction: AI Is Reshaping Software Testing

Software testing is undergoing a profound, AI-driven transformation. In the past, testers had to hand-write large numbers of test cases, maintain fragile automation scripts, and inspect execution results one by one. Today, as large model capabilities mature, a brand-new proposition emerges: how can we truly embed AI into the entire testing lifecycle and build a "quality system" that isn't constrained by tools or engines?

Understanding this transformation requires a bit of historical perspective. Software testing has evolved from manual testing (1950s–1970s) through structured testing methodologies (1980s), the rise of automation testing frameworks (1990s–2000s, exemplified by Selenium and JUnit), the "shift-left" testing driven by Continuous Integration/Continuous Delivery (CI/CD) (2010s), and now to the AI-driven intelligent testing era. The core tension at every stage has been between "test quality" and "delivery speed." AI's involvement fundamentally changes this dynamic: traditional automation can only execute paths predefined by humans, whereas AI-driven testing can autonomously generate test paths from the semantics of requirements, transforming the expansion of test coverage from linear growth to exponential expansion.

This article is based on the core ideas of an AI + automated testing course. Its value lies not in cataloging tools but in proposing a clear architectural viewpoint: using Skill plus CLI as the core hub, supporting two upward-facing deployment forms—platformization and digital employees. This line of thinking is instructive for any team looking to introduce AI into the testing domain.

Intelligent Testing Platform: A Front-and-Back-End System Covering the Entire Testing Lifecycle

An intelligent testing platform is essentially a complete front-and-back-end system that covers the key stages of testing work:

• Automation orchestration: Execute an automation flow, configurable with just a few drag-and-drop steps;
• Intelligent parameter recognition: Parameters that need to be passed in are automatically identified by the system, with no manual entry required;
• Test case generation: AI generates test cases with a single click, and they can still be manually edited and corrected afterward;
• UI testing and test point analysis: View test point analysis results directly, with a clear structure at a glance;
• Execution result visualization: Every operation, every request, and every screenshot of the interface can be fully traced back.

It's worth emphasizing the design philosophy of "AI-generated, human-editable." This philosophy aligns closely with the industry's Human-in-the-Loop principle.

Human-in-the-Loop (HITL) is an engineering paradigm that embeds human judgment into the AI decision loop, originating from the fields of cybernetics and reinforcement learning. In reinforcement learning, HITL uses human feedback signals (RLHF) to train models to align with human preferences—OpenAI used exactly this method to train InstructGPT and GPT-4. In engineering practice, HITL more broadly refers to setting human review checkpoints at key nodes of an AI automation flow to prevent model hallucinations or logical errors from escalating into irreversible production incidents—this is especially critical in testing scenarios, because an incorrect testing conclusion could directly allow a defect into the production environment.

In its 2024 Hype Cycle, Gartner clearly noted that fully autonomous AI testing is still in the "Peak of Inflated Expectations," while human-machine collaborative Augmented Testing is entering the "Plateau of Productivity." Leading AI testing tools such as Mabl, Testim, and Applitools all adopt similar strategies: AI is responsible for generating and maintaining initial versions of test cases, while testers focus on business logic validation and boundary condition review. This division of labor not only dramatically compresses the maintenance costs of traditional UI automation scripts caused by page changes (typically accounting for 30%–50% of total testing investment), but also preserves humans' final judgment over business semantics. AI is not meant to replace testers, but to shoulder the tedious first-draft work and return the final judgment and correction authority to humans. This human-machine collaboration model is precisely the most pragmatic path for engineering AI into practice today.

The Core Hub: Skill + CLI Endows AI with Boundless Capabilities

The most critical design in this entire system is using Skill and CLI as the core hub.

The CLI (Command Line Interface) is the bridge through which AI interacts with the external world. The course offers an incisive statement: "With the CLI, AI can possess boundless capabilities." The logic behind this is that a large model itself is merely a language engine—the true boundary of its capabilities depends on how many tools it is connected to.

The technical foundation of this mechanism is the large model's Tool Use / Function Calling capability. After OpenAI introduced the Function Calling feature in 2023, this paradigm quickly became the infrastructure standard for engineering AI Agents. Its principle is: developers pre-register a set of function JSON Schema descriptions with the model (including function name, parameter types, and functional descriptions); during inference, if the model judges that a function needs to be called, it outputs structured JSON conforming to that Schema rather than free text; the host program parses it, executes the real operation, and injects the return value into the conversation context. This mechanism forms a complete closed loop of "perception → planning → execution → feedback," solving the large model's core limitation of "only being able to talk, not act," and represents the key technical leap for AI Agents to evolve from conversational assistants into autonomous execution systems.

Worth noting is the ongoing evolution of Function Calling capabilities: Anthropic's Claude uses a similar mechanism but calls it Tool Use; Google DeepMind built a code interpreter into Gemini as a native tool; and in 2024, OpenAI further introduced Parallel Function Calling, allowing the model to call multiple tools simultaneously in a single inference, dramatically boosting Agent execution efficiency. Notably, the accuracy of Function Calling depends heavily on the quality of the Schema description—the more precise the description and the more sufficient the examples, the higher the probability the model selects the correct tool, giving rise to the emerging engineering subfield of "Prompt Engineering for Tools." The CLI is the natural extension of this mechanism—wrapping any command-line tool into a Skill that can be invoked by AI, enabling AI to call testing frameworks, operate browsers, make requests, and read screenshots, turning "being able to talk" into "being able to act."

But capability alone is not enough. Two core problems must be solved in tandem:

The First Problem: How Are Capabilities Integrated

In mainstream Agent frameworks such as LangChain, AutoGen, and Semantic Kernel, a Skill or Tool is defined as a callable unit that encapsulates a specific function, comprising three elements: input/output specifications, a functional description, and execution logic. The model decides when to call and how to pass arguments by reading the Skill's description. Once a team accumulates dozens or even hundreds of Skills, how to manage Skill registration, version control, and context passing evolves into an engineering problem requiring a unified scheduling hub—that is, how to integrate these scattered capabilities into a workflow in an orderly fashion rather than having them operate in isolation.

The Second Problem: How Are Capabilities Custom-Developed

Off-the-shelf Skills will always leave some scenarios uncovered. Therefore, a team must not only know how to "use" Skills, but also possess the ability to custom-develop tools for AI. In the testing domain, Skills can encapsulate atomic capabilities such as browser operations, API requests, screenshot comparison, and log parsing, and the boundaries of these capabilities are determined by the team's own business scenarios. This means testing teams need to transform from "script writers" into "capability builders."

It is precisely because this mechanism is built on the abstraction layer of Skill + CLI that it can be engine-agnostic—not limited to any specific tool or engine. This corresponds to the design philosophy of "Hexagonal Architecture" (Ports and Adapters Architecture) in software architecture, proposed by Alistair Cockburn in 2005. Its core idea is to place the application's business logic at the center, interacting with external systems through abstract "ports" (interface definitions), while concrete technical implementations are plugged in as "adapters." This allows infrastructure such as databases, UI frameworks, and third-party services to be independently replaced without affecting the core logic. In an AI testing system, this means that when Playwright is replaced by a next-generation testing engine, or GPT-4 is replaced by a more powerful model, the Skill invocation logic and test orchestration flow do not need to be rewritten, greatly reducing the cost of technical migration. In an era where AI capabilities make generational leaps every few months, an engine-agnostic architecture is essentially buying insurance against an uncertain technical future. The same quality system can run in different AI programming environments—whether Codex-type tools or various cloud-based workbenches. Engine agnosticism is an important hallmark of architectural maturity.

Two Deployment Forms: Platformization and Digital Employees

Once the Skill system is built, it can manifest in two distinctly different yet complementary forms.

Platformization: A Manageable System with an Extremely Low Barrier

The core value of platformization is "lowering the barrier to use." Its design goal is: even someone with no understanding of AI needs only to know the operational path of "I have test points → select an agent to generate test cases." As for how the Skills are invoked or how the models are configured, they need not concern themselves at all.

Platformization also brings two key capabilities: monitorability and manageability:

• Monitorability: Know in real time how many AI digital employees are currently running and which agent is executing what task;
• Manageability: Uniformly edit and maintain all Skills, with support for cross-project reuse; you can even mix and match models of different costs according to task importance, achieving a precise balance between effectiveness and investment.

"Mixing and matching models of different costs according to task importance" corresponds to the engineering practice known in the industry as LLM Routing or Model Cascading. LLM Routing is essentially a cost-optimization strategy that dynamically selects models based on task complexity; Martin Fowler likens it to the API gateway pattern in microservices architecture—a unified entry point with rule-based dispatching. Common routing dimensions used in practice include: input token count, task classification labels (classification/generation/reasoning), and historical accuracy. Open-source projects such as RouteLLM and LiteLLM have already engineered this capability. Take the testing scenario as an example: simple tasks such as parameter extraction and format conversion cost about 1/20 of a flagship model when using a GPT-3.5-level model, while complex test logic reasoning and root-cause analysis of anomalies require calling flagship models. Establishing a clear task-model correspondence matrix is a key cost-control measure for advancing AI testing from proof of concept to large-scale deployment; reasonable routing can reduce overall inference costs by 60%–80%.

Digital Employees: AI as Virtual Team Members

The same mechanism can also be packaged into a "digital employee" form. From a technical standpoint, digital employees are typically built on a Multi-Agent System (MAS) collaboration framework. Multi-Agent systems are an important paradigm of distributed AI, decomposing complex tasks into multiple specialized agents that collaborate to complete them. Microsoft's AutoGen introduced the concept of "conversational agents," where agents collaborate through message passing and support human intervention at any stage; CrewAI, on the other hand, centers on "role-playing," endowing each agent with clear responsibilities, tool sets, and goals.

In engineering practice, multi-agent systems face three core challenges: First is the "orchestration consistency" problem—how to avoid state conflicts when multiple agents execute concurrently, which the industry typically solves with Event Sourcing or distributed lock mechanisms. Second is the "hallucination propagation" problem—a single agent's erroneous output may be taken as fact by downstream agents and used for continued reasoning, creating an error-amplification effect; Microsoft Research's "Debate between LLMs" approach is an effective countermeasure. Third is the "runaway cost" problem—every node in a multi-agent chain consumes tokens, and without fine-grained routing, the inference cost of a single complex testing task could reach 5–10 times that of a single-agent solution. The solutions to these three challenges are precisely what constitute the core barrier distinguishing a mature AI testing platform from a toy-grade demo.

In testing scenarios, the typical collaboration hierarchy of digital employees is: a task-planning agent (responsible for understanding requirements and breaking down test points), an execution agent (invoking Skills such as Selenium or Playwright to complete page interactions), and a supervisory agent (comparing expected results against actual screenshots, responsible for result verification and exception handling). The three have clear divisions of labor and mutually verify one another, significantly improving accuracy over single-agent solutions in complex testing scenarios. A typical workflow is: receive the test requirement description → automatically analyze test points → invoke Skills to generate cases → drive the browser to execute → aggregate results and generate a report, with no human intervention throughout. At this point, AI is no longer a passive tool but more like a virtual member who can independently take on testing tasks, autonomously completing the closed loop from analysis to execution. The fundamental difference from the platformization form lies in "autonomy": the platform is humans driving AI, while digital employees are AI proactively taking on tasks.

The coexistence of the two forms reflects the elasticity a mature AI system should have: unified underlying capabilities, with the upper-layer delivery method chosen on demand.

Conclusion: The Essence of a Quality System Is Abstraction and Decoupling

Reviewing this entire line of thinking, the most profound insight lies in abstraction and decoupling:

1. Capability-layer decoupling: Standardize and modularize AI capabilities through Skill + CLI;
2. Engine agnosticism: The core system is not bound to any specific AI tool or model, possessing cross-platform portability;
3. Flexible delivery forms: Platformization lowers the barrier, while the digital-employee form enhances autonomy;
4. Controllable cost: Model mix-and-match strategies optimize the input-output ratio.

For testing practitioners, the core competitiveness of the future is no longer proficiency in writing scripts for a particular framework, but rather understanding how to build capabilities for AI, orchestrate workflows, and manage systems—this is a role transition from "executor" to "system designer." AI will not replace testers, but testers who master this system will be far ahead of peers still hand-writing scripts.

Key Takeaways

Key Takeaways

Key Takeaways