Three Stages of AI LLM Testing: The Testing Transformation from Assertions to Evaluations

AI is reshaping the boundaries of software testing. Over the past six months, the industry's understanding of "AI + testing" has evolved from vague to clear: using AI to assist testing is now a thing of the past, AI-led testing is becoming mainstream, and testing (evaluating) AI applications themselves is the third piece of the puzzle that must be filled in. Based on a tech talk on Bilibili, this article outlines the three-stage evolution of AI LLM testing and takes a deep dive into why traditional assertion-based testing fails in the AI era, and how an evaluation system can be put into practice.

1. The Three-Stage Evolution of AI Testing: From Assistance to Leadership to Evaluation

According to industry observations from the Bilibili creator, the AI testing system can be divided into three progressive stages: testing with AI, testing alongside AI, and testing AI itself.

Testing with AI refers to using AI tools to assist humans in completing testing tasks and improving efficiency. This stage was a hot topic just over half a year ago, but leading Chinese tech giants have already moved past it. Practices at Tencent, Alibaba, ByteDance, and DeepSeek^Ad in the AI testing space show that mere "AI assistance" is no longer sufficient for current needs.

Testing alongside AI enters the stage of human-machine division of labor. Most testing work—requirements analysis, test case design, system exploration, code writing, automated regression, and more—can be handed over to AI, with humans stepping back into a collaborator role. The creator notes that this model is being pushed by leading companies as the new standard practice.

Testing AI itself is the newest and most complex stage. As applications like AI customer service, AI video generation, and AI fortune-telling proliferate, these systems integrate AI capabilities internally, and their testing approaches and processes are fundamentally different from traditional software. The creator calls this "AI application testing" and predicts that this type of demand will continue to grow as AI software becomes more widespread.

2. Why Must We Shift from "Assertions" to "Evaluations"?

Traditional software testing relies on assertions: does the status code equal 200, does the field content match, does the return value exist. Assertions give binary results—pass or fail—deterministic and stable. But the output of AI large models is inherently probabilistic.

From a principles perspective, most current mainstream large models are based on the Transformer architecture, and their output is essentially a probabilistic choice—the same question may yield different answers. From a results perspective, even when the output is deterministic (say, the number 2), its representation can vary widely—Chinese uppercase numerals, Arabic numerals, English words, binary, hexadecimal, and so on, with vastly different styles. A tiny change in model parameters or prompts can cause significant shifts in results.

Therefore, the creator emphasizes: testing AI can no longer use simple, blunt assertions; it should adopt scoring-based evaluation. This is not a new invention but an international convention—just as the college entrance exam selects students by score, and medicine uses rating scales to assess depression or tumors, complex systems are naturally suited to expressing "how good" or "how bad" with continuous scores.

One detail: evaluation does not entirely replace testing. Evaluation gives a score; testing draws the pass line. The creator explains: scores are continuous values (0 to 1), but they can also be discretized into 0 and 1 to represent pass or fail. Whether an AI product can be released still requires a clear judgment based on an evaluation score threshold.

The Transformer architecture is the foundational neural network structure of current mainstream large language models (such as GPT and DeepSeek), proposed by Vaswani et al. in the 2017 paper "Attention Is All You Need." Its core is the Self-Attention mechanism, which allows the model to dynamically attend to all positions in the input sequence when processing each word, thereby capturing long-range dependencies. Unlike recurrent neural networks (RNNs), Transformers can process the entire sequence in parallel, greatly improving training efficiency. Through large-scale text pre-training, the model learns probabilistic relationships between words, so when generating answers it does not retrieve fixed results but samples word by word based on context—this is precisely why the same input can produce different outputs, the fundamental reason AI output is probabilistic and traditional assertion-based testing fails.

3. Version Pinning: The Dual Constraint of AI Evaluation

The volatility of AI evaluation places higher demands on version management. The creator proposes that two versions must be pinned:

The system-under-test version: including the model used, input parameters, prompts, context, and so on. Any change to any of these means the tested version has changed. For example, if you used DeepSeek during testing and later switched to GPT, the results need to be re-evaluated and reinterpreted.

The evaluation version: including the dataset, assertions, prompt templates, scoring thresholds, and other testing instruments. If the dataset or threshold changes, results are likewise not comparable.

Only when both versions are pinned simultaneously can results be comparable. If only one version is replaced, you can do a cross-sectional comparison (e.g., comparing model quality under the same evaluation instruments); if both change at once, the results are completely independent and must be reinterpreted.

4. What to Evaluate? Three Layers of Objects and a Progressive Path

The creator divides evaluation objects into three layers of increasing complexity:

Large Language Models (LLMs): evaluate the correctness, completeness, safety, and stability of generated content. Although ordinary companies don't train models, they can use evaluations for model selection, prompt optimization, and even verifying the results of fine-tuning smaller models.

Knowledge Bases (RAG): the principle is to retrieve relevant materials and then answer based on those materials. Evaluation focuses on retrieval accuracy and generation faithfulness—whether the generated result is based on the specified reference materials, i.e., the "faithfulness" metric.

Agents: involving task planning, tool use, and task progression. Evaluation needs to cover tool selection, task orchestration, knowledge base usage, and other aspects. The creator points out that the three layers are progressive rather than a choice of one: an agent often contains a knowledge base and a large model, making it the most complex.

The learning path is therefore clear: test large models first, then knowledge bases, and finally agents.

RAG (Retrieval-Augmented Generation) is a technical architecture that combines an external knowledge base with a generative model. Its workflow typically has two steps: first, retrieve relevant document fragments from the knowledge base based on the user's question (retrieval stage); then feed these fragments as context along with the original question into the large language model, letting the model generate an answer based on the given materials (generation stage). RAG can effectively mitigate large model hallucinations (fabricating facts) and outdated knowledge problems, and is widely used in enterprise knowledge Q&A, customer service, and other scenarios. When evaluating a RAG system, one must separately examine retrieval quality (e.g., recall, precision—whether the correct materials were found) and generation faithfulness (whether the generated content is strictly based on the retrieved materials rather than the model improvising); these two types of metrics together determine the reliability of the RAG system.

5. Tool Selection: Why Promptfoo Is Recommended

The creator compared several AI evaluation tools and highlighted Promptfoo. Reasons include: CLI command-line driven, meaning it can be called by AI itself, realizing the potential of "AI evaluating AI"; supports writing test cases in YAML, friendly to those with no coding background; supports batch invocation and Python custom extensions; and has been acquired by OpenAI, with a clear ecosystem outlook.

For scenarios requiring metric research, other academic-style tools can be used; for production monitoring of online agents, there are corresponding monitoring tools. But most testers' goal is acceptance judgment—evaluating whether software can be released—and Promptfoo covers all three layers of evaluation (large models, knowledge bases, agents) in this scenario, making it a relatively balanced choice.

Promptfoo is an open-source AI evaluation and red-teaming tool, initially focused on comparing and optimizing prompts, later expanded into an evaluation platform covering large models, RAG, and agents. Its core design philosophy is configuration as code: users define test cases, evaluation metrics, and model configurations through YAML files, then batch-execute evaluations via the command line (CLI) and generate visual reports. Being CLI-driven means it can be automatically invoked by scripts or AI agents, enabling automated evaluation in continuous integration. Promptfoo has built-in assertion types (such as contains, similarity, LLM scoring, etc.) and supports Python custom scoring functions, making it easy for teams to extend metrics based on business needs. In 2024, Promptfoo was acquired by OpenAI, indicating that its technical approach has gained mainstream recognition, with a clear outlook for ecosystem integration.

6. Why Performance Testing Experience Becomes an Advantage

The creator uses an analogy to help understand AI evaluation: performance testing likewise doesn't care about single-instance correctness; it finds problems through a set of data under specific conditions, and requires two rounds of comparison before submitting a problem. This is highly similar to the logic of AI evaluation—both rely on comparison and scoring rather than binary assertions.

Those with performance testing experience can transfer metric thinking such as concurrency rate and RPS to AI evaluation metrics like faithfulness and recall, significantly lowering the barrier to understanding. The creator admits that most testers haven't systematically studied performance testing, so recommends supplementing this knowledge as a bridge to understanding AI evaluation.

Key metrics in performance testing such as concurrency rate (the number of requests processed simultaneously per unit time), RPS (Requests Per Second), response time, and error rate reflect the system's overall performance under a specific load, rather than the correctness of a single request. Performance testing typically requires multiple rounds of comparison (e.g., before and after optimization, under different configurations) to locate bottlenecks, which is consistent with the logic of discovering problems in AI evaluation through comparing scores across multiple sets of data. Transferring the metric thinking of performance testing to AI evaluation—for example, analogizing RPS's "throughput" to evaluation's "sample coverage," and response time's "distribution" to score "distribution"—can help testers more quickly understand the methodology of continuous scoring, statistical comparison, and threshold determination, lowering the cognitive barrier to transitioning from traditional testing to AI evaluation.

Conclusion

The transformation of AI testing is not a minor patch at the tool level but a paradigm shift at the methodological level: from assertions to evaluations, from single versions to dual version pinning, from a single object to three progressive layers. The creator uses 30 minutes of theoretical groundwork to emphasize that directional understanding is more important than tool operation—being able to clearly discuss these concepts in interviews and grasp the general direction at work is far more valuable than mechanically running a demo. For testing practitioners, catching up on the lesson of "testing AI itself" is no longer optional.

One-line Summary

AI testing is moving from assistance to leadership, and AI applications themselves must be evaluated; assertion-based testing fails, and scoring-based evaluation with dual version pinning becomes key.

Paragraph Summary

Based on a Bilibili tech talk, this article outlines the three-stage evolution of AI LLM testing: testing with AI, testing alongside AI, and testing AI itself. It points out that traditional assertion-based testing fails due to the probabilistic nature of large model output, and that we must shift to continuous scoring-based evaluation from 0 to 1—but evaluation does not replace testing, as threshold judgment is still needed for release. AI evaluation requires pinning both the system-under-test version and the evaluation version to ensure comparability. Evaluation objects fall into three progressive layers: large language models, knowledge bases (RAG), and agents. For tools, Promptfoo is recommended for its CLI support, YAML test cases, Python extensions, and acquisition by OpenAI. Performance testing experience can be transferred to AI evaluation to help understand metric thinking. The article emphasizes that directional understanding is more important than tool operation, and testing practitioners must catch up on the lesson of "testing AI itself."