The AI Software Testing Paradigm Shift: A Complete Guide from Assertions to Evaluations

The Three Pillars of AI Testing: From Assistant to Driver

With sustained investment from AI-leading companies like Tencent, Alibaba, ByteDance, and DeepSeek, AI adoption in the testing field has moved from concept to practice. This article draws on real-world industry experience to outline three core testing paradigms in the AI era — and the critical shift from "testing" to "evaluation."

Today's AI testing landscape can be broken down into three modes:

• Testing with AI: Using AI tools to assist testing and improve efficiency (now becoming a baseline capability)
• Testing alongside AI: Humans and AI each playing distinct roles, with AI handling the majority of testing work
• Testing AI itself: Dedicated testing of software systems that have AI features built in

Here's a key industry signal: leading companies have already entered the "AI-driven testing" phase — where core activities like requirements analysis, test case design, system exploration, code writing, and automated regression are all handled by AI.

Why Traditional Assertions No Longer Work

The Fundamental Gap Between Determinism and Probability

At the heart of traditional software testing is the assertion mechanism: results are either right or wrong, pass or fail. In API testing, for example, you assert a status code or a response field — outputs are deterministic, stable, and predictable.

In the world of large language models (LLMs), this logic breaks down completely. There are two reasons why:

From a technical standpoint, mainstream generative LLMs are built on the Transformer architecture, which involves probabilistic sampling at inference time. The same question can yield different answers — that's an inherent property of the model.

From an output standpoint, even a "correct" answer can take many forms. The number "2" could appear as the Chinese characters "贰" or "二", the English word "two", the Arabic numeral "2", or a representation in a different numeral base. Any fine-tuning of model parameters or prompts can shift the output.

This is why evaluating AI isn't about judging "right or wrong" — it's about using a scoring mechanism to express "how good" or "how bad" a response is. Much like how exam scores convey relative performance rather than a binary pass/fail, scoring is the right tool for handling complex, open-ended outputs.

How Evaluation Relates to Testing

AI evaluation doesn't throw testing out the window — it reframes it. Evaluation handles the scoring problem, while testing defines the passing threshold for those scores. Just as college entrance exams have cutoff scores and medical screening has diagnostic thresholds, AI products need clear criteria: can the system go live? Can it be released? These decisions are made by comparing evaluation scores against a defined threshold.

Scoring and assertions can also be converted into each other. AI evaluation often uses continuous values between 0 and 1 (e.g., 0.1, 0.9, 0.999), but you can also use the discrete values 0 and 1 to represent the traditional "fail" and "perfect score" — bridging back to assertion-style logic when needed.

The Core Requirement for AI Evaluation: Fixing Two Versions

AI evaluation demands stricter version control than traditional testing — and it requires fixing two separate dimensions:

The System Under Test (SUT) Version

The SUT version isn't just a software build number. It also includes:

• The model type in use (e.g., GPT-4, DeepSeek)
• The model's parameter configuration (temperature, top_p, etc.)
• The prompt strategy applied
• Any contextual information provided

Changing any one of these constitutes a version change. For example, switching from DeepSeek to GPT means you're no longer on the same version — and you'll need to re-evaluate from scratch.

The Evaluation System Version

This is a dimension unique to AI evaluation: how you're testing also needs to be versioned:

• The test dataset used
• The evaluation criteria applied
• The threshold values set

Only when both versions are fixed does a side-by-side comparison mean anything. If you only swap one (e.g., evaluating different models with the same methodology), you can compare model quality. If both change at once, the results are incomparable.

A useful analogy: AI evaluation closely resembles performance testing. Performance testing also isn't concerned with simple right/wrong outcomes — it discovers issues under specific conditions using datasets, and requires multiple comparison runs to draw conclusions.

Three Layers of AI Evaluation Targets

Many people assume AI evaluation is just about stacking up metrics (accuracy, recall, etc.) — but the first step is defining what you're evaluating, because the goals and metrics differ entirely depending on the target.

There are three common evaluation targets, and they follow a progressive relationship:

1. Large Language Models (LLMs)

Evaluating the correctness, completeness, safety, and consistency of generated content. Applicable scenarios:

• Model selection: Comparing different models on performance, speed, and capability
• Prompt optimization: Validating the impact of prompt improvements
• Fine-tuned model evaluation: For example, verifying the effectiveness of locally fine-tuned smaller models like Qwen variants

2. Knowledge Bases (RAG)

RAG systems retrieve relevant materials first, then generate answers grounded in those materials. Key evaluation dimensions:

• Retrieval accuracy
• Recall performance
• Faithfulness (whether the generated output is actually supported by the retrieved source material)
• End-to-end pipeline coherence

3. AI Agents

Agents are the most complex evaluation target, encompassing:

• Task planning capability
• Tool selection strategy
• Task orchestration logic
• Knowledge base integration
• Underlying LLM performance

Customer service agents, data agents, and domain-specific agents in travel, legal, and medical contexts are becoming increasingly widespread.

Recommended learning path: These three aren't alternatives — they're a progression. Master LLM evaluation first → then RAG → then Agents.

Tool Selection: Why Promptfoo Stands Out

A solid methodology needs the right tooling. Promptfoo is the recommended go-to tool for most teams:

Core Advantages

• CLI-driven architecture: Command-line and YAML-based workflows mean the tool itself can be called by AI, enabling "AI evaluating AI" scenarios
• Support for human evaluation golden standards: Write test case assertions in YAML for zero-code test packaging
• Strong extensibility: Supports batch invocation and Python customization
• Industry validation: Acquired by OpenAI, lending top-tier credibility to its quality

Differentiated Selection Guidance

• Academic research and model development: Consider specialized tools focused on detailed metric data
• Production environment monitoring: User-facing agents are better served by production tracing tools
• Acceptance testing (the majority of testing scenarios): Promptfoo is the best fit, covering all three evaluation layers — LLMs, RAG, and Agents

Methodology Matters More Than Tool Mechanics

Hands-on tool operation and live demos are valuable — but they all rest on a methodological foundation. When you're facing directional uncertainty, revisiting these core frameworks will be far more useful:

• The paradigm shift from assertions to evaluations
• The two-version fixation principle
• The progressive relationship between the three evaluation targets

Understanding these fundamentals is what lets you establish yourself in the emerging field of AI testing and navigate a rapidly evolving technical landscape.