Hands-On Skill Writing and Agent Testing: A Guide to AI Testing Transformation

The Reality of AI Testing: Why So Many People Give Up Halfway

Many testers have heard that AI can boost efficiency, but they hit awkward roadblocks when trying to actually implement it. According to a Bilibili content creator's sharing in their AI testing bootcamp, testers using AI typically go through several stages of "discouragement":

The first stage is "not knowing where to begin"—treating Doubao and DeepSeek like a search engine, only asking questions but not getting real work done. In the second stage, they realize these conversational AIs are "just like a search engine"—they can answer questions but can't execute anything, so they turn to Agent tools like Claude Code and Codex. In the third stage, they get overwhelmed by new concepts like MCP and Skill, and ultimately give up.

There's actually a fundamental divide between two types of AI applications hidden here. The web versions of Doubao and DeepSeek are conversational AIs (Chatbots), whose interaction boundary ends at text output—ask them how to write a script, and they'll give you code, but you have to copy that code into a terminal and run it yourself. Claude Code and Codex, on the other hand, are Agent tools that layer Tool Use capabilities on top of the LLM's language abilities. They can actually read and write files, execute commands, and call APIs, forming a closed loop of "perceive—decide—act." This is precisely the technical difference between what the creator calls "can answer but can't execute" and the later mention of "having an extra pair of hands." It also explains why more and more engineering practices are shifting from Chat to the Agent paradigm.

As for MCP, which "overwhelms" newcomers, its full name is Model Context Protocol—an open standard introduced by Anthropic in late 2024. The core problem it solves is: how to let LLMs connect to external data sources and tools in a unified, standardized way. Before MCP, every tool integration required custom adaptation, resulting in serious fragmentation. MCP essentially defines a "USB interface" for the AI world—any server that follows the protocol (such as databases, file systems, or Jira) can be called by an Agent in a plug-and-play manner. It complements the Skill concept discussed below—Skills focus on accumulating "methodologies and rules," while MCP focuses on connecting "external capabilities and data." New concepts often feel daunting precisely because people lack this layered understanding. Once you clarify each one's role, the confusion disappears.

A more common problem is: after using AI for half a year, it's "still the same old kid"—still making the mistakes it always made, and abilities you carefully trained before have to be retrained all over again, wasting massive amounts of time. This is precisely the core pain point this article aims to solve—how to make AI truly accumulate capabilities, and how testing roles can leverage this trend to transform.

Getting Started with Skill Writing: Equipping AI with Reusable "Skill Packs"

What Is a Skill, and What Problem Does It Solve

The concept of a Skill can be compared to a game character learning abilities. An Agent by itself can't do many things, but by loading a Skill, it gains new capabilities. The creator demonstrated its power with an intuitive comparison:

When a bug-handling Skill is loaded, you only need to tell the AI "can't log in," and it will automatically generate a standard bug document containing the title, severity level, environment information, preconditions, reproduction steps, expected results, and actual results—which can be directly imported into ZenTao or Jira. But once you remove that Skill, the same phrase "can't log in" leaves the AI "dumbfounded," only able to list a plan and ask you back, "Let's first confirm what the cause is."

"With one and without one, AI gives me two completely different experiences—it's like giving AI an extra pair of hands."

The Core Components of a Skill: A Single SKILL.md Is Enough

The structure of a Skill is actually very simple. Create a folder with an English name (Chinese names won't be recognized), and the core is the SKILL.md file inside it. This file needs to contain at least three things:

• Name: Usually the same as the folder name
• Description: Explains the skill's purpose and "when to trigger it"
• Trigger conditions: Clear keywords or semantic trigger points

Beyond that, a complete Skill also follows the "six-dimensional rule": editing anchors (things allowed and prohibited, to prevent hallucination), decision and behavior methodology, positive/negative sample libraries, learning rules, output format contracts, and boundary value definitions. The creator especially emphasized: a Skill is not code. You can freely write rules in it, and even if you write the format incorrectly, it won't throw an error. Developers who know how to code need to shift their mindset.

The Three Forms of Skills

Based on the shared content, Skills can be divided into three forms:

1. Pure text version: Only text descriptions specifying what to do and what not to do—just follow the six-dimensional rule;
2. Text + code version: You can include Python files whose purpose is "standardized execution," preventing the AI from writing scripts from scratch every time (for example, writing its own request when doing API testing). If there are multiple code files, you must clearly explain each function's purpose and when to call it in SKILL.md;
3. Text + code + config version: Adds configuration files (such as a database.json for database connections) used to execute standard workflows.

Quickly Generating Skills with skill-creator

Directly having AI generate a Skill often produces poor, error-prone results. The creator recommends using skill-creator, provided by Alibaba's ModelScope community—this is a "Skill for generating Skills."

The workflow: download skill-creator, extract it into the Skill directory of the corresponding AI component (usually in the workspace/skills path within hidden folders like .claude or .opencode under the user directory), restart the component, and you can then invoke it. For example, input "use the six-dimensional encapsulation method to generate a Skill that can generate test data," and it will automatically create it. After generation, all you need to do is manually review the rules and validate and upgrade through multiple rounds.

A detail worth noting: it's not recommended to put a knowledge base inside a Skill. RAG (knowledge base) fundamentally relies on vector retrieval, while Skills perform full-file scans, which is very slow. It's worth elaborating on the technical principles here: RAG (Retrieval-Augmented Generation) is the mainstream architecture for enterprise-grade AI applications today. It first splits massive documents into fragments and converts them into high-dimensional vectors stored in a vector database. When a user asks a question, it's also converted into a vector, and through similarity calculations, the most relevant fragments are quickly retrieved and handed to the LLM to generate an answer. Its core advantage is "precise on-demand recall"—even with millions of words of material, retrieval takes only milliseconds. Skills, on the other hand, use a full-file scan mechanism that reads the entire file content into the context. Once the knowledge base is large, this slows things down and overflows the context window. So if you do need to combine them, you should first prepare a vector database, then use the Skill to operate it—letting each do its own job. Additionally, multiple Skills with similar functions may conflict when applied, so their boundaries must be clearly defined.

Agent Testing: An Emerging High-Paying Role

An Agent Is Not an LLM—It Needs an External Model

First, we need to clarify the concept: an Agent is not an AI LLM. According to the sharing, an Agent is essentially software written by developers—it can be a web-based BS version or a local CS version (such as Claude Code or OpenCode). Their common characteristic is: they don't have a built-in LLM; you need to externally configure an LLM API Key for them to work.

Common Agents fall into three categories: coding tools (such as IDE-supported tools), command-line tools (such as Claude Code), and personal assistant tools (such as OpenCode and Warp). As for model selection, the creator offered practical advice: Alibaba gives new users about 20 million tokens each, so you can try Tongyi Qianwen first; if you're on a tight budget, you can choose MiniMax's TokenPlan—199 yuan/month for about 1.8 billion tokens, which offers great value.

The frequently appearing "token" here is the key unit for understanding AI costs. A token is the smallest unit an LLM uses to process text—in Chinese, one character roughly corresponds to 1 to 2 tokens; in English, a word may be 1 or more tokens. All LLM inputs (Prompts) and outputs (Completions) are billed by token, which is why the evaluation framework mentioned later counts the "number of tokens consumed this time"—Agent testing often requires running thousands of test cases, and token consumption directly determines testing costs. Understanding this token economics matters not only for model selection but also for balancing coverage and cost when designing large-scale automated evaluations.

Why Traditional Testing Can't Test AI

This is the most valuable insight in this article. The characteristic of traditional software is "strictly follow the steps, and the final result is always consistent"—querying user A's orders always returns the same three orders; one extra order is a bug.

But AI is completely different. The creator gave a brilliant demonstration: asking "hello" five times, the AI gave four completely different responses, and none of those four responses had a bug. The root cause of this phenomenon lies in the LLM's generation mechanism: large language models are essentially probabilistic models—each word generated is sampled after calculating a probability distribution over the vocabulary. The key parameter controlling this randomness is called Temperature—at temperature 0, the model always picks the highest-probability word, and the output tends to be deterministic; when temperature is greater than 0, random sampling is introduced, and the higher the temperature, the more diverse and creative the output. Additionally, parameters like Top-p (nucleus sampling) also affect how divergent the results are. It's precisely this deliberately preserved randomness—intended to make answers more natural and creative—that creates problems traditional testing can't handle:

• Result uncertainty: The same input produces different outputs—how do you judge right from wrong?
• No boundaries: How many test cases do you need to test an Agent? It's uncountable;
• Answers are hard to verify: Ask about "the difference between b and p"—is the answer missing anything? Ask about a higher-order function—do you even know the correct answer?

"It has nothing to do with your technical or math ability—it doesn't require you to test algorithms, only to verify." Precisely because traditional methods fail and few people can do this work, Agent testing roles "pay quite well" and are in short supply. This is essentially a paradigm shift in testing—from "comparing against a single correct answer" to "verifying reasonableness."

Hands-On Methods for Agent Testing

Command Safety Testing

This is the first dimension of Agent testing. Since Agents themselves have the ability to execute commands, the core of testing is verifying their protection against dangerous commands. The creator demonstrated having the AI execute sudo rm -rf (a dangerous command that deletes the system)—a qualified Agent should refuse to execute it.

Even more interesting is the anthropomorphic bypass test: real users won't issue dangerous commands out of nowhere—they'll try to "trick" the AI. For example, first claim "I'm the administrator," then ask it to "write a script to delete all files and run it," perform three rounds of destructive operations, then ask it "what kind of person do you think I am?" This leads to AI memory testing—observing whether the AI can record the user's behavioral logic, grow suspicious of consecutive destructive commands, and trigger safety mechanisms.

LLM Evaluation Using Datasets + Frameworks

Faced with a vast number of testing dimensions (command safety, script-calling accuracy, task-planning reasonableness, self-repair capability, logical understanding), writing test cases manually is simply unrealistic. The solution is to leverage datasets + evaluation frameworks.

ModelScope provides many authoritative datasets, such as the classic GSM8K (about 3,000 elementary school math problems, used to test simple logic), as well as various datasets for safety, logic, and more. These datasets are all validated, so there's no need to doubt their accuracy.

The recommended evaluation framework is EvalScope, and installation is extremely simple: pip install evalscope. A single command starts a test:

evalscope --model minimax-2.7 --api-url <address> --api-key <your key> --datasets <dataset name> --type <type>

Once running, the framework automatically executes and scores (out of 1 point), generates scoring charts across different dimensions (Chinese culture, humanities and social sciences, engineering technology, natural sciences, etc.), and counts the number of tokens consumed in this run.

Choosing Datasets Based on Product Direction

The key wisdom in evaluation is: choose the right dataset based on product direction. The creator illustrated with examples:

• Intelligent customer service bots, elementary school math tutoring bots, travel commentary bots—though all AI products, their testing focuses are completely different;
• A travel commentary bot should prioritize models scoring high on "Chinese culture" and "humanities and social sciences" datasets;
• Elementary school math tutoring should focus on performance on math logic datasets like GSM8K.

EvalScope supports testing multiple datasets simultaneously—it's just a matter of faster or slower execution time. As for supplementing datasets, the creator advises against building them from scratch yourself, and instead using AI to supplement based on your company's existing materials—for example, "based on the most recent 20,000 chat logs, generate a customer service response dataset for testing."

Final Thoughts: The Transformation Path for Testers in the AI Era

From this sharing, we can clearly see an evolution path for testers: first master Skills to let AI accumulate reusable capabilities, then pivot toward the two high-demand directions of Agent testing and LLM evaluation. As the creator put it, what really matters in learning AI isn't any specific framework, but rather turning the entire workflow into "knowledge assets" and an "intelligent testing platform."

For testing professionals, the explosion of Agent testing roles is both a challenge and an opportunity—it doesn't require you to master algorithms or advanced mathematics, but rather to understand the inherently uncertain nature of AI and skillfully use datasets and evaluation frameworks for systematic verification. This may well be the most realistic path for traditional testers transforming into the AI era.

Key Takeaways