AI Testing in Practice: A Complete Guide to Three Types of AI Tools and the New Paradigm of Test Development

The Polarization Dilemma in the Testing Industry in the AI Era

As we enter the AI era, a strange phenomenon has become widespread across the testing industry and the entire IT sector: perceptions of AI are severely polarized. Some believe AI is omnipotent and can replace the vast majority of work; others believe AI offers limited help in actual work and can't solve the real problems.

Behind this polarization lies a lack of basic understanding of how technology evolves and iterates. Many people first encounter AI through two entry points—DeepSeek and Claude Code (what the original text called "OpenCloud" should refer to the Claude Code ecosystem). DeepSeek is a series of open-source large language models released by the Chinese company DeepSeek. With extremely low training costs and reasoning capabilities on par with world-class models, it drew global attention in early 2025 and became the starting point for many domestic users to encounter AI. Claude Code, on the other hand, is a command-line programming assistant built by Anthropic around its Claude series of models, designed to understand codebases directly and perform modification tasks within a terminal environment. The two represent two typical entry points for AI adoption today: the former leans toward general dialogue and reasoning, while the latter is deeply embedded in development workflows. So whenever big news breaks, someone asks, "Such-and-such technology just came out—does it have a disruptive impact on testing?"

But the real issue is this: if an entire industry undergoes disruptive change because of a single technology, it proves the industry itself may have a problem. AI brings not simple replacement, but a redistribution of capabilities. Understanding this is the prerequisite for mastering AI tools.

Three Types of AI Tools: Each Has Its Strengths, None Is a Master Key

The AI tools on the market are dazzling: Claude Code, Codex, Cursor, OpenCode, ByteDance's Trae, Qcode… Many people can't figure out which one to actually use. In fact, these tools can be grouped into three major categories, and each category has a direction it excels at—no single tool can solve every problem.

Category One: Personal Assistant Tools

The strength of these tools lies in integration with social/collaboration tools and handling lightweight tasks. Their defining characteristic is a low degree of engineering complexity—you don't need to build complex engineering projects; you might just ask the AI to gather information, write a simple document, or engage in everyday conversation.

A typical use case for this category is remote work: your computer is at the office, but you're at home. By connecting to remote tools like DingTalk, Feishu, or WeCom, you can complete work tasks just by chatting with the AI on your phone. Vendors like Qwen, NetEase, and Tencent have all released similar open-source products, and users can choose according to their needs.

Category Two: Command-Line (CLI) Geek Tools

These tools have no graphical interface—pure command-line interaction. They're called "geek tools" because they abandon the human-computer interaction concepts designed over the past few decades. Their core philosophy is: "We don't need to interact with an interface; we only need to talk to the AI." The typical representatives are Claude Code and its open-source alternative OpenCode.

These tools are extremely practical in server operation scenarios. The article demonstrates OpenCode in action: after connecting to a Linux server via SSH, you can directly ask "How many Java environments are on the server?" or "Help me configure the environment variables," and the AI can identify the system's state, provide an installation and configuration plan, and execute it.

Its biggest advantage is that the AI is embedded within the system and understands the real state of the current server. This stands in stark contrast to standalone conversation tools like DeepSeek or Doubao—the latter don't know your server, and while the commands they give may look convincing, you don't know what will happen when you run them, often requiring repeated copy-paste-and-debug cycles. In addition, OpenCode has built-in free models (such as MiniMax) that work right after installation, with no extra API configuration needed.

Category Three: AI IDE Programming Tools

This is the transformation product of the traditional engineering field. The "old-school programming" of the past using IDEA or PyCharm required hand-writing every line of code; in the AI era, you just describe your goal and the AI can write large blocks of code. Even non-professionals can complete development without knowing how to program. Representative tools include Cursor and Trae, and PyCharm and VSCode also have AI plugins such as Tongyi Lingma.

Why You Should Master One Tool From Each Category

The author gives a key conclusion: you'll use all three categories, but you don't need to learn every single one—just master any one from each category. Because in the AI era, the gaps between tools are rapidly leveled out—when Claude Code designs a new feature, OpenCode's developers quickly catch up. These tools are all essentially about opening a dialogue box to interact with an AI, and the ways of using them are largely the same.

But why master one from each category rather than just learning one? The answer is that different scenarios call for different tools, and the experience varies dramatically:

• Writing code but not seeing any interface at all and not checking any of the code is obviously inconvenient—so AI programming requires the "Claude Code + IDE" combination, using a visual interface to review the generated code;
• When operating a server that has no graphical interface, a GUI-based personal assistant tool becomes a burden—the pure command-line OpenCode is more suitable;
• Having to open separate software to view generated documents and reports also interrupts the workflow.

The author also shares his own practical combination: leave code writing to Claude Code (widely recognized as first-tier in engineering capability), while pairing it with domestic tool Trae for reviewing, checking, and modifying code.

Generating Test Cases Is Easy—The Hard Part Is Making AI Understand the Business

The article uses a highly instructive demonstration to shatter a common misconception. The author executed the same instruction—"Generate 8 API test cases based on the interface information and save them to an MD file"—using both OpenCode (the free MiniMax model) and Claude Code (a paid model), and both tools completed it easily.

The conclusion is clear: which tool you use to generate test cases is not the problem at all—a single sentence gets it done, with zero technical difficulty.

So what's the real difficulty? The author cuts straight to the point:

"Whether it's API test cases or functional test cases, the most important thing is how the AI understands your business system."

In reality, an API doc might list two parameters while three are actually used in the call; a requirements doc might mention two buttons while there are actually three. As a tester, you can't demand that product managers and developers hand you a document perfected down to every detail. What limits testers has never been the AI tool itself, but how to solve problems when documentation is incomplete.

This also involves a key technology—RAG (Retrieval-Augmented Generation). Because a large model's training data has a cutoff date and cannot cover an enterprise's internal private data, RAG significantly improves the accuracy and timeliness of answers by first retrieving relevant documents from a knowledge base and then feeding the retrieved results to the model as context. In testing scenarios, RAG allows the AI to understand the business system based on the latest API docs and requirements docs—which precisely addresses the core difficulty of "making the AI understand the business."

A deeper issue is the data source: AI needs material to generate test cases, so where does real interface information come from? Copy-pasting from the F12 developer tools is extremely inefficient, and scripts recorded by JMeter are hard to feed directly to the AI. This is the real threshold when it comes to actual implementation.

AI Test Development: Core Competencies Under the New Paradigm

To solve the data-source problem, the author demonstrates the thinking behind a self-developed tool: have Claude Code invoke a custom "recording" Skill that launches a special browser dedicated to recording interface, page, and snapshot information, in preparation for subsequent document generation and test case generation.

This brings up two key concepts:

What Exactly Is a Skill

Many people think learning AI just means chatting with an AI, and installing a Skill is enough. But the author clarifies: a Skill is essentially just a description file—a piece of instructional information shown to the AI, telling it what the skill is supposed to do and what commands to run. It's not magic; Claude Code, OpenCode, and Trae all have their own Skill folders. What actually does the work behind a Skill is a separately developed tool program.

It's worth noting that Skills and tool invocations often rely on a unified connection standard—MCP (Model Context Protocol). This is an open standard proposed by Anthropic in late 2024 to unify how large models connect with external tools and data sources. Before MCP appeared, every AI tool had to develop separate interfaces to connect with external systems, and the standards were chaotic. By defining a universal communication specification, MCP allows models to invoke resources like databases, APIs, and file systems in a standardized way—similar to a "USB port" for the AI field. This is why the author emphasizes that capabilities like MCP and Skills are universal across tools—the underlying protocol is consistent; the tools are just different shells.

From Test Development to AI Test Development

Developing these tools that help AI work better used to be called "test development," and now it's called "AI test development." The author emphasizes that even with AI, you still need the mindset and programming skills of test development, for two reasons:

1. The code the AI writes will throw errors and have problems, and you have to be able to correct it;
2. The AI is trained in advance and doesn't know about the latest technical changes—for example, a framework may have just released a fix for a bug two hours ago, but the AI's training data still reflects the old version, so it will write code containing the already-fixed bug.

Therefore, in the AI era, learning automation and test development has changed in two ways: learning is faster (what used to take three or four months now takes 20 days to a month), but the focus has also shifted—it's no longer about mastering every line of code, but about understanding the implementation principles of automation and the design thinking behind frameworks, and then using AI to complete the code writing.

The Essence of Automation: From "Having People Write It" to "Having AI Write It"

The author offers a sharp critique of the "record scripts to do automation" approach. He points out that recorded scripts have been rejected from start to finish, and the fundamental reason is the data problem: what's recorded is fixed data that can't be executed repeatedly (for example, canceling an order—once you've canceled it, running it again is meaningless). Systems change quickly, and scripts are a burden whether modified by people or by AI.

The real difficulty in automation is filling in the "missing arms and legs":

• Where is the token extracted from? How does it correspond to the token returned upon login?
• The order number should be dynamic—how do you implement dynamic automated execution?
• The dependencies between interfaces—how does the AI know about them?

In the traditional automation era, we used YAML/CSV configuration to let "people" complete automation with zero code; the AI era goes a step further—even people no longer write it, the AI does. This requires creating a specification that the AI can understand, with extensive customization tailored to AI invocation. This is what AI automation is really about.

AI Levels the Gaps, and Also Creates New Opportunities

The article ultimately arrives at a profound industry judgment: AI is a technology that levels gaps.

In the past, many people couldn't do performance testing well, getting stuck on details like environment setup, memorizing commands, and server operations. Now, AI helps you set up the environment and generate commands (such as quickly viewing "the program consuming the most CPU"), turning "memorizing a 1,000-word essay" into "memorizing 100 words while still being able to flip through the book." The technical gaps between people with three, five, or eight years of work experience are rapidly leveled by AI.

So where is the new gap? It lies in who can use AI better and understands AI more deeply—the use of agent tools, the development of AI testing tools, the use of models, and the development of platforms. At the same time, an entirely new field has emerged: how to test AI itself. As more and more products undergo AI transformation, Skills, prompts, RAG, MCP, agents, and large-model fine-tuning all need to be tested and evaluated—this has become a new, high-value direction.

Conclusion: Don't Chase Hype, Grasp the Fundamentals

The core point the author repeatedly emphasizes is: don't chase the hype. When others say Claude Code is hot, you learn Claude Code; when they say Codex is hot, you learn Codex—you'll never keep up, and whatever you learn will be obsolete within a month.

"What you should learn is the fundamentals of these technologies—the essence never changes amid all the variation. Skills, MCP—these things are all consistent; the tools just look different."

As long as you skillfully master any one tool, capabilities like Skills, MCP, and Agents are universal and not confined to any single tool. This is the learning direction truly worth investing in during the fast-iterating AI era.

Key Takeaways