Three Phases of AI Testing: The Mindset Shift from Assertions to Evaluation

The Three Evolutionary Phases of Testing in the AI Era

As AI penetrates ever deeper into software development, the testing profession is undergoing a structural transformation. A Bilibili creator who specializes in test development (formerly a Python-focused developer now writing projects in JS) shared his latest observations on AI testing frameworks: ideas that felt hazy and unclear just six months ago are now becoming increasingly sharp and well-defined.

He organizes the testing landscape of the AI era into three levels:

• Using AI to test: Leveraging AI to assist with testing and improve efficiency. This was the hottest topic in testing six months ago, but the author considers it "already a thing of the past."
• Testing alongside AI: Humans and AI each playing their respective roles. Tencent, Alibaba, and ByteDance — three Chinese companies with significant AI investment — along with DeepSeek^Ad as it pushes into engineering research, are driving "AI-led testing," where AI can handle most of the work: requirements analysis, test case design, system exploration, code writing, and automated regression.
• Testing AI itself: Moving beyond the question of human-AI division of labor, the focus shifts to how to test software with embedded AI features — such as AI customer service, AI fortune-telling, and AI video generation. The thinking and process for this type of testing differ fundamentally from traditional software testing, and the industry now calls it "AI application testing."

Why "Evaluation" Instead of "Testing"

The author's core argument is that testing AI cannot follow the traditional assertion-based approach — it must shift toward "evaluation." This isn't a concept invented by any one person in China; it's international convention. Both Chinese and global communities have consistently used the term "Evaluation" when assessing artificial intelligence.

The logic is straightforward. Traditional software testing relies on assertions: assert the status code, assert the returned data, assert field values. The result is always binary — "right" or "wrong," "pass" or "fail" — with consistent outputs and stable, reliable assertions. But the output of large language models is probabilistic, making simple equality checks fundamentally unsuitable.

The author explains this probabilistic nature from two angles:

Inherent Probabilism at the Architecture Level

Most mainstream generative large models are based on the Transformer architecture, which inherently involves a probabilistic selection process during output generation. As a result, large models have "probabilism baked into their DNA" — ask the same question, and you may get a different answer.

The Transformer is a neural network architecture proposed by Google in 2017 in the paper Attention Is All You Need. Nearly all mainstream large language models today — GPT, Claude, DeepSeek, and others — are built on it. Its core mechanism is "Self-Attention," which captures dependencies between any positions in a sequence. When generating text, the model does not directly output a single determined word; instead, it computes a probability distribution over the entire vocabulary and selects from it using a sampling strategy (such as Top-K, Top-P sampling, or temperature-scaled Softmax). The higher the temperature parameter, the more random the sampling and the more diverse the output; at a temperature of 0, the output approaches determinism, though it remains a special case of the probabilistic mechanism rather than a truly deterministic one. This "sample by probability" design is precisely what allows the same question to produce very different answers across different calls — it's an architectural feature of large models, not a flaw.

Diversity at the Output Level

Even when the result is definitive (say, the number "2"), it can be expressed in countless forms: uppercase, lowercase, English words, Arabic numerals, binary, hexadecimal, ASCII, and more — each potentially in different styles. Factor in that any small change to model parameters or prompts can cause dramatic shifts in output, and it becomes clear why simple equality assertions fall short. An exact match might occasionally be correct, but there are "tens of millions" of ways things can be unequal. A blunt assertion is clearly inadequate.

The author offers an apt analogy: you can't simply label a person a "good person" or a "bad person" — that kind of judgment is severely limiting. Assessments of medical conditions (cancer, tumors, depression, IQ) are all expressed as scores. Just as the college entrance exam uses scores to differentiate students, AI that mimics human behavior should be evaluated and scored accordingly.

Evaluation Doesn't Mean Abandoning Testing

A common misconception is that shifting to evaluation means abandoning testing altogether. The author explicitly rejects this.

Evaluation is fundamentally about "scoring" — expressing how good or bad a result is through a number. But above the evaluation layer, there still needs to be a testing step, which is like drawing a "passing threshold" for those scores. Just as college admissions have cutoff scores, and tumor or depression diagnoses have diagnostic thresholds, AI products still need clear go/no-go decisions before launch or release.

In practice, AI evaluation typically uses continuous values between 0 and 1 (0.1, 0.2, 0.9, 0.999…), but you can also use the two discrete values 0 and 1 to map to traditional assertions of "fail" and "full score," achieving discretization and simplification. In real-world projects, both continuous-value evaluation and traditional assertion-based testing have their place.

Dual Version Locking: A Strict Requirement for AI Evaluation

Because evaluation results are inherently ambiguous and variable, the author emphasizes that AI evaluation demands stricter version management than traditional testing — specifically, two versions must be locked simultaneously:

• The version under test: The version of the system being tested, defined by the model in use, the parameters passed in, the prompts used, and the context provided. Any change to any of these — say, switching from DeepSeek to GPT — means the version under test has changed, and results must be re-evaluated from scratch.
• The evaluation version: What you're using to run the test — the dataset, assertions, prompts, and thresholds. If the dataset or threshold changes, the results naturally differ.

The key rule: when only one version is replaced, you can make meaningful cross-comparisons (e.g., comparing model quality across different models using the same evaluation approach, or comparing threshold effects on the same system). But if both versions change simultaneously, the comparison loses its validity and you're left with a completely independent, standalone evaluation result.

The author also draws a parallel for those with performance testing experience: performance testing similarly doesn't fixate on pass/fail; instead, it uses a set of data collected under specific conditions to surface problems, and requires multiple rounds of comparison before conclusions can be submitted. This mirrors the AI evaluation mindset closely.

What to Evaluate: Three Progressive Layers — LLM, Knowledge Base, and Agent

The author corrects a common mistake: AI evaluation doesn't start by piling on metrics like recall rate and faithfulness. The right approach is to first identify what you're evaluating, because different subjects have entirely different goals and metrics. He organizes evaluation targets into three progressive layers — not parallel options, but a sequence:

Layer 1: Large Language Models (LLMs)

This layer primarily evaluates the correctness, completeness, safety, and stability of LLM-generated content. It's essential testing for model vendors like DeepSeek; for regular companies, it's useful for model selection (which model is better, faster, smarter), prompt optimization, and — for well-resourced teams — evaluating small models after fine-tuning.

Layer 2: Knowledge Bases (RAG)

A knowledge base first retrieves relevant information, then generates answers based on that information. Evaluation focuses on each step: how accurate is the retrieval? How faithful is the generation (does the output actually reflect the retrieved material)? Metrics like recall rate and faithfulness primarily apply at this layer.

RAG (Retrieval-Augmented Generation) is a technical paradigm that combines external knowledge bases with large models. The basic workflow: when a user asks a question, the system first retrieves relevant document chunks from the knowledge base (typically via vector similarity search), then passes the retrieved content as context — along with the original question — to the LLM, which synthesizes an answer. RAG addresses the knowledge cutoff dates and hallucination issues of large models, and is the dominant approach for enterprise AI customer service and internal knowledge Q&A.

Two key metrics apply when evaluating RAG systems: Recall measures whether the knowledge base retrieves all relevant documents — low recall means "relevant content wasn't found"; Faithfulness measures whether the model's generated answer is genuinely grounded in the retrieved material, rather than "free-styling" and hallucinating. If either step fails, final answer quality suffers significantly — which is why layered evaluation, rather than only examining the final output, is essential.

Layer 3: Agents

Customer service agents, data agents, and knowledge base agents are already fairly common; agents for travel, legal, and medical domains are on the horizon. Agents involve task planning, tool usage, and task execution, making their evaluation the most complex — it encompasses tool selection, task orchestration, knowledge base usage, and at its core, LLM usage.

Complexity and knowledge depth increase with each layer, forming a clear learning path: master LLM evaluation first, then knowledge base evaluation, then agent evaluation.

An AI Agent is an AI system capable of autonomously perceiving its environment, making plans, calling tools, and executing multi-step tasks — distinct from a single-turn LLM query. A typical agent architecture comprises three core components: a planning module (decomposing complex goals into executable steps), tool calling (search engines, code executors, database queries, external APIs), and a memory module (short-term context and long-term knowledge storage).

The evaluation complexity of agents far exceeds that of a single LLM layer, because errors accumulate across steps — an early planning mistake can send all subsequent tool calls in the wrong direction, and each tool call itself can also fail. Evaluation must consider: task completion rate, accuracy of tool selection, reasonableness of the task path (were redundant steps taken), and correctness of the final result. This multi-step, multi-tool nature renders traditional single-turn assertions completely ineffective; the entire execution trajectory must be evaluated.

What to Use: Choosing Evaluation Tools

The final question is tooling. The author lists several AI testing tools and gives a strong recommendation for what he describes as a "Promptfoo-style" command-line tool, citing several reasons:

• PyTest-style: Very friendly for those with PyTest automation experience, aligning with code-driven testing habits.
• CLI-driven: Because it supports command-line operation, the tool itself can be invoked by AI. "This tool is used to evaluate AI, and it can itself be used by AI" — the future scenario of "AI evaluating AI" has significant potential.
• Supports human evaluation: In AI evaluation, human assessment is the "gold standard" (because the current industry consensus is that AI is still not as intelligent as humans). The tool supports writing test case assertions in YAML format.
• Highly extensible: Supports batch calls and custom Python extensions.
• Acquired by OpenAI: Its excellence was recognized by an even more excellent company.

The author also offers differentiated selection advice: those doing academic metric research (e.g., model vendors) may prefer metric-focused tools; those running online agents who need production monitoring and tracing should look at the appropriate tooling there. For most testing professionals, however, evaluation is about determining whether a product is ready to ship — it has an acceptance-testing character — and the recommended tool is well-suited for this, covering all three evaluation layers: LLMs, knowledge bases, and agents.

Final Thoughts

The author repeatedly emphasizes that these theoretical, directional concepts are "the most dry but also the most valuable" part of the entire lesson. Subsequent tool operation and script execution are all concrete implementations built on this conceptual framework. Once you understand the mindset shift from assertions to evaluation, the dual version locking methodology, and the three-layer progressive evaluation structure, you'll be able to hold a substantive conversation in interviews even without hands-on practice — and have a clear direction when real tasks come your way.

For testing professionals weighing their career direction heading into 2026, this framework offers a grounded reference point: AI hasn't eliminated the testing role — it has redefined it. The job is shifting from judging "right or wrong" to measuring "how good."