AI Large Model Testing in Three Phases: The New Paradigm from Assertions to Evaluation

AI Is Redefining Software Testing

Software testing is undergoing a profound transformation. A testing-focused content creator on Bilibili points to a clear industry trend: AI hasn't just changed how software is developed — it has fundamentally reshaped how it's tested. He candidly shares that he came from a test engineering background with a focus on Python, and has now started writing projects in JS: "The industry is changing, and learning can't stop."

Half a year ago, the industry conversation around AI testing was still centered on "AI-assisted testing for efficiency gains" — but that framing has already become outdated. Tencent, Alibaba, and ByteDance — the three domestic companies pushing hardest on AI — along with DeepSeek^Ad, which has shifted from model research toward engineering, are all driving testing into a new era.

The author proposed a three-system framework for AI-era testing a year ago, and it has only grown clearer since: Testing with AI, Testing alongside AI, and Testing AI itself. Understanding these three progressive layers is the essential foundation for breaking into AI testing roles.

Three Testing Paradigms: With AI, Alongside AI, and of AI

Testing with AI and Testing alongside AI

"Testing with AI" refers to using AI to assist humans and improve testing efficiency — the earliest form of AI in testing, and now largely a thing of the past. It has been replaced by "Testing alongside AI," where AI takes the lead.

In this model, testers are no longer fully in charge. Instead, humans and AI each handle their respective responsibilities. The author emphasizes that most of the work — requirements analysis, test case design, system exploration, code writing, and automated regression — can be handled by AI, with humans focusing only on critical decision points.

Testing AI Itself: The Rise of AI Application Testing

The third paradigm, "Testing AI itself," is a thread the author planted a year ago that has now come to fruition. As more and more software integrates AI capabilities internally — AI customer service, AI fortune-telling, AI video generation, and so on — testing these AI-powered features has become an entirely new challenge.

The thinking and process behind this type of testing are completely different from traditional software testing, which is why it's treated as its own discipline, often called AI application testing. This is the core focus of this article.

From Assertions to Evaluation: A Fundamental Shift in Testing Mindset

Traditional software testing relies on assertions: a result either passes or fails — there's no third option. In API testing, you can assert status codes, response data, and field values. Outputs are deterministic, and assertions are stable and reliable.

But large language models shatter that determinism. The author explains the probabilistic nature of AI outputs from two angles:

First, at a technical level, mainstream large models use the Transformer architecture, which inherently involves probabilistic selection during output generation. LLMs are probabilistic by nature — the same question may yield different answers.

Second, at the output level, even when an answer is conceptually fixed (say, the number "2"), it can appear in many forms: Chinese characters, Arabic numerals, binary, hexadecimal, and more. On top of that, any minor change in model parameters or context can drastically alter the result. Simple equality assertions no longer apply.

This is why AI testing has adopted the internationally recognized term Evaluation. The logic mirrors medical diagnosis or college entrance exams: you can't simply declare a person "good" or "bad," "smart" or "not smart" — you measure the difference through scoring. Testing AI, which is designed to emulate human intelligence, calls for exactly the same approach.

Evaluation Doesn't Replace Testing

It's worth clarifying that evaluation doesn't fully replace testing. Evaluation answers the question of "how many points does this score?" while testing sets the passing threshold — just as college admissions have cutoff scores and medical diagnoses have clinical thresholds.

AI evaluation typically uses continuous scores between 0 and 1 (e.g., 0.1, 0.9, 0.999), but you can also use the discrete values 0 and 1 to simulate traditional assertions: 0 means fail, 1 means full marks. This discretization allows traditional testing and AI evaluation to be used interchangeably in practice.

---

Background: Why LLMs Are Probabilistic

The Transformer architecture is the foundational structure behind today's mainstream large language models (such as GPT and DeepSeek), introduced by Google in the 2017 paper Attention Is All You Need. Its core mechanism — self-attention — allows the model to weigh the relevance of every token in context when generating each new token. During generation, the model doesn't output a single deterministic word; instead, it computes a probability distribution over the entire vocabulary and selects an output through sampling or greedy strategies. The temperature parameter controls how flat or sharp this distribution is: higher temperature means more randomness; temperature near 0 approaches deterministic output. This is the fundamental reason why the same question can yield different answers in different conversations — and why simple equality assertions break down when applied to LLMs.

---

Version Pinning: The Strict Demands of AI Evaluation

Because evaluation results are fuzzy and prone to variation, version management in AI testing is even more rigorous than in traditional testing. The author stresses the need to pin two versions simultaneously:

The version under test: This isn't just a software version number — it includes the model being used, the parameters passed in, the prompts, and the context. Any change to any of these means the version under test has changed. For example, switching from DeepSeek to GPT means the results must be re-evaluated from scratch.

The evaluation version: This refers to the testing methodology itself — the dataset, assertions, prompts, and thresholds used. Just as taking the Jiangsu provincial exam versus the national exam will naturally yield different scores, changing your evaluation setup changes your results.

Meaningful comparison requires changing only one version at a time. Keep the evaluation method fixed and swap the model, and you can compare model quality. Keep the system under test fixed and change the threshold, and you can compare the effect of different cutoffs. If both versions change simultaneously, the results have no comparative value — they become completely independent evaluations.

The author also offers a useful analogy: AI evaluation closely resembles performance testing. Neither is primarily concerned with right or wrong; both discover problems by running a dataset under specific conditions, and both require multiple rounds of comparison rather than a single judgment call. Those with performance testing experience will find this logic intuitive.

What to Evaluate: Three Progressive Layers — LLM, Knowledge Base, and Agent

Many people assume AI evaluation simply means plugging in a set of metrics (like recall or faithfulness). The author corrects this misconception: you must first identify what you're evaluating, because the goals and metrics differ entirely depending on the object.

Layer 1: The Large Language Model (LLM). The focus is on the correctness, completeness, safety, and stability of generated content. For companies like DeepSeek, this is a mandatory evaluation; for most other companies, it's used for model selection, prompt optimization, and validation after fine-tuning.

Layer 2: The Knowledge Base (RAG). RAG works by retrieving relevant documents first, then using them to generate an answer. Evaluation therefore focuses on each stage of the pipeline — whether retrieval is accurate, and whether the generated output has a corresponding reference. The faithfulness metric assesses whether the answer is strictly grounded in the specified source material; recall also primarily applies here.

Layer 3: The Agent. Customer service agents, data agents, legal agents, medical agents — these will only grow more prevalent. Their operation involves task planning, tool use, and task execution, making them the most complex to evaluate.

These three layers are progressive, not mutually exclusive: an agent typically has a knowledge base attached, and the knowledge base depends on an underlying LLM. The recommended learning path follows this order: start with LLMs, move to knowledge bases, then tackle agents.

---

Background: What Is RAG?

RAG (Retrieval-Augmented Generation) is one of the most widely adopted approaches for deploying LLMs in enterprise settings. The workflow has two steps: first, the user's question is converted into a vector and used to retrieve the most relevant document chunks from a pre-built knowledge base (typically a vector database); then, both the retrieved content and the original question are fed into the LLM, which generates an answer "based on the materials" rather than relying solely on knowledge learned during training. RAG's strength is that it enables low-cost integration of private knowledge and reduces the risk of model hallucination — but it also introduces two independent failure points: the retrieval stage may find the wrong content or nothing at all, and the generation stage may ignore the retrieved content and improvise anyway. This explains why RAG evaluation needs to separately assess "recall" (did retrieval find the right content?) and "faithfulness" (does the generated answer truly reflect the retrieved material?), rather than just checking whether the final answer is correct.

---

Tool Selection: Why Promptfoo Is Worth Recommending

Finally, there's the question of "what to test with." The author surveys several AI testing tools and highlights one PyTest-style evaluation framework in particular — Promptfoo — for several reasons:

It's driven by CLI and YAML, which means it can be invoked by AI itself — paving the way for "AI evaluating AI" scenarios in the near future. It also supports human evaluation (in AI evaluation, human judgment is the gold standard, since AI still isn't smarter than humans), batch execution, and custom Python extensions. Perhaps most notably, it has been acquired by OpenAI, a recognition of its quality by one of the industry's leading organizations.

The author also offers differentiated recommendations: for academic metric research, specialized evaluation tools may be more appropriate; for agents serving live users, production monitoring tools are better suited; but for most testers, whose goal is to determine whether a product is ready to ship (essentially an acceptance testing mindset), Promptfoo is the most practical choice — and it covers all three layers: LLMs, knowledge bases, and agents.

---

Background: What Is Promptfoo?

Promptfoo is an open-source LLM evaluation framework whose core design philosophy is to describe test cases in YAML configuration files and execute evaluations via the CLI — similar in spirit to PyTest in the Python ecosystem. Its basic structure consists of three components: providers (the models or APIs under test), prompts (prompt templates), and tests (test cases and assertion rules). Assertion types include exact match, regex, semantic similarity, and calling an external LLM as a "judge" for scoring — flexibly covering everything from discrete assertions to continuous scoring. Because the entire evaluation workflow can be triggered via CLI, Promptfoo integrates naturally into CI/CD pipelines for automated regression evaluation during model version updates. This engineering-first design is its core advantage over academic evaluation tools that primarily rely on notebooks.

---

Final Thoughts

The author repeatedly emphasizes that the seemingly dry theory covered here — evaluation mindset, version pinning, layered evaluation objects, tool selection — is actually the most valuable part of the entire AI testing framework. All the hands-on tooling and scripting that follows is built on top of this methodology. Even without immediately diving into practice, understanding these directional concepts is enough to give you a real edge in interviews or on the job. For anyone looking to enter the hot AI testing field in 2026, mastering the shift in thinking from assertions to evaluation is an unavoidable first lesson.