AI Large Model Testing: A Complete Guide to the Three-Layer Evaluation Framework

From Testing to Evaluation: A Paradigm Shift in Software Quality Assurance for the AI Era

The software industry is undergoing a profound role transformation. Testers are moving into development, and developers need to understand testing — as one Bilibili content creator who transitioned from test development to JS projects put it bluntly: the industry is changing too fast to stop learning. In the testing space specifically, AI adoption has rapidly evolved from "AI-assisted testing efficiency" half a year ago to "AI-driven testing" — and even spawned an entirely new direction: testing AI itself.

This creator categorizes AI-era testing into three modes: testing with AI (using AI to boost efficiency), testing alongside AI (humans and AI each handling their respective responsibilities), and testing AI itself (validating software that contains AI functionality). The first two have largely become reality as major players like Tencent, Alibaba, ByteDance, and DeepSeek^Ad push toward engineering-level adoption. And as AI-powered customer service bots, AI fortune tellers, AI video generators, and countless other applications proliferate, ensuring the quality of these AI applications has become an unavoidable challenge for testing professionals.

Why AI Requires Evaluation, Not Assertion

At the heart of traditional software testing is the assertion — a result is either right or wrong, either passing or failing, with no third option. You assert status codes, assert returned fields, assert data content. Outputs are deterministic, results stable and reliable. But AI large models have fundamentally shattered this certainty.

The creator explains the source of probabilistic behavior from two angles. At the architectural level, mainstream large models are based on the Transformer architecture, which inherently incorporates a probabilistic selection mechanism during output — the same question may yield different answers, and this is baked into the model's very DNA. At the output level, even a seemingly definite result (like the number "2") can be expressed in countless forms: Chinese characters, lowercase letters, Arabic numerals, binary, hexadecimal, ASCII, and more. On top of that, any minor change in model parameters or context can trigger dramatically different results.

This is precisely why simple "equal" or "not equal" assertions no longer apply — the matching scenario may be correct, but there are tens of millions of possible non-matching scenarios. The industry has therefore adopted the internationally accepted concept of "evaluation." Just as you can't judge a person as simply good or bad, and diagnosing illness requires a scoring system rather than a binary conclusion, evaluating AI — which mimics human behavior — requires a scoring approach to measure "how good" or "how poor" a result is.

The probabilistic output mechanism of the Transformer architecture stems from the "sampling" strategy used during its decoding phase. When generating each token, the model doesn't simply select the highest-probability option — instead, it samples randomly from a probability distribution. The degree of randomness is controlled by the "Temperature" parameter: higher temperature produces more random and diverse outputs; lower temperature (approaching 0) produces more stable and convergent outputs. Additional parameters like Top-p (nucleus sampling) also influence the output distribution. This means that even with identical input prompts, each call to the model may yield different responses — this is expected behavior by design, not a defect. For this reason, traditional single-run, single-point assertion testing cannot reliably capture the true quality of probabilistic systems; multi-sample, statistical distribution-based evaluation is required instead.

Evaluation Doesn't Mean Abandoning Testing

One important misconception to clear up: the AI evaluation space hasn't abandoned testing entirely in favor of pure evaluation. The creator emphasizes that "evaluation" refers to the concept of "scoring," while "testing" is about setting a passing threshold for those scores.

Just as college entrance exams have admission cutoffs, and tumor or depression diagnoses have scoring thresholds, determining whether an AI product can go live ultimately still requires a clear pass/fail judgment — and that judgment is based on testing the evaluation scores against a defined threshold.

In terms of how scores are assigned, the AI field typically uses continuous values between 0 and 1 (e.g., 0.1, 0.2, 0.9, 0.999). But continuous values can also be discretized: using 0 for failure and 1 for full marks effectively returns to the traditional pass/fail assertion logic. In real-world projects, you'll encounter both continuous scores and binary 0/1 outcomes — you need to be comfortable with both.

Version Pinning: AI Evaluation Is More Rigorous Than Traditional Testing

Traditional testing has the concept of version numbers — which version is being tested, which version has a bug, what changed between versions — and test reports are only accountable to a specific version. Because AI evaluation results are inherently ambiguous and may fluctuate, version management is even more demanding: you need to pin two versions.

The system-under-test version refers to the version of the system being evaluated, including which model is used, what parameters are passed, what prompts are used, and what context is added — any change to any of these constitutes a version change. For example, switching from DeepSeek to GPT means the version has changed, and results need to be re-evaluated and re-interpreted.

The evaluation version refers to the methodology used for testing, including the dataset, assertions, prompts, and thresholds. If the dataset or threshold changes, the results will differ too — just as the same student might score differently on the Jiangsu provincial exam versus the national exam.

Only by fixing one version while varying the other does a comparison carry meaningful value — fixing the evaluation methodology while swapping models lets you compare model quality; fixing the system under test while varying the threshold lets you compare threshold effectiveness. If both versions change simultaneously, the comparison loses its validity, and results can only be interpreted as entirely new, independent data points.

The creator also suggests using performance testing as an analogy for understanding AI evaluation: performance testing similarly isn't about simple right/wrong, but rather about discovering issues through a set of data under specific conditions, requiring at least two rounds of comparison before conclusions can be submitted. This "comparative" mindset closely mirrors AI evaluation, making it easier to pick up for those with performance testing experience.

What AI Evaluation Actually Measures: A Three-Layer Progressive Framework

The prerequisite for evaluation is clarity about what you're evaluating — different subjects require different purposes and metrics. The creator breaks down common evaluation subjects into three layers, and these layers are progressive rather than mutually exclusive.

Layer 1: Large Language Models (LLMs). The primary focus is on the correctness, completeness, safety, and stability of generated content. For companies like DeepSeek whose product is the model itself, this is a mandatory testing area. For typical companies, it's useful for model selection (which is better, faster, smarter), prompt optimization, and validating the effectiveness of fine-tuned smaller models.

Layer 2: Knowledge Bases (RAG). The principle is to first retrieve the right materials, then answer questions based on those materials. Evaluation therefore needs to cover each stage: whether retrieval is accurate, faithfulness (whether generated results are supported by corresponding reference materials), and metrics like recall rate primarily apply at this layer.

Layer 3: Agents. This encompasses task planning, tool use, and task execution. Customer service agents and data agents are already fairly common, and agents for travel, legal, medical, and other domains will continue to grow. Agents have the highest complexity because they simultaneously involve tool selection, task orchestration, knowledge base usage, and ultimately still call upon large models at their core.

Complexity increases across these three layers, and the learning path becomes clear accordingly: master LLM evaluation first, then knowledge base evaluation, then agents. The creator notes that the hands-on portion of this particular session only covers Layer 1 (LLMs), with the latter two left for future sessions.

RAG (Retrieval-Augmented Generation) is currently one of the most mainstream architectures for enterprise AI deployment. The core idea is to split private knowledge bases (such as company documents and product manuals) into text chunks and store them as vectors. When a user asks a question, the system first retrieves the most relevant chunks from the knowledge base, then incorporates those chunks as context into the prompt, allowing the large model to generate a response based on the retrieved materials. The advantage of this approach is that it injects up-to-date private knowledge without retraining the model. However, it introduces new failure points: inaccurate retrieval leaves the model with insufficient material to work with, and even with accurate retrieval, if the model fails to faithfully follow the source material when generating its response (the "hallucination" problem), quality will still suffer. RAG evaluation therefore requires independently measuring both retrieval quality (e.g., recall rate, precision) and generation quality (e.g., faithfulness, answer relevance).

Tool Selection: Why Promptfoo

The final key question is which tools to use. The creator strongly recommends Promptfoo, for several reasons.

It's a PyTest-style tool, making it highly accessible for those with a coding background — especially anyone with PyTest automation experience. It's driven by CLI and YAML — which is an interesting detail: this tool for evaluating AI supports CLI, which means AI can in turn use the tool itself, laying the groundwork for future "AI evaluating AI" scenarios.

On evaluation methodology, human evaluation remains the current "gold standard" (since the industry consensus is still that AI hasn't surpassed human intelligence). But this tool supports writing test case assertions in YAML for zero-code encapsulation, while also supporting batch invocation and Python custom extensions — offering strong extensibility and promising prospects. Most importantly, it has been acquired by OpenAI, providing validation from a top-tier AI company.

The creator also offers differentiated selection advice: for academic metric research, choose tools focused on metric data; for production monitoring of user-facing agents, choose appropriate monitoring tools; but for the majority of testing professionals whose goal is "acceptance and release," Promptfoo is the better fit — and it covers all three evaluation layers: LLMs, knowledge bases, and agents.

Promptfoo's core workflow is: define prompt templates, test case inputs, and assertion rules in a YAML file, then run them in batch via CLI, scoring each model output and generating a summary report. Its assertion types offer broad coverage — from deterministic assertions like string matching and JSON Schema validation, to the LLM-as-a-judge mode, where another LLM acts as a "judge" to evaluate output quality. The latter is a concrete implementation of the "AI evaluating AI" concept. In this judge mode, users simply describe evaluation criteria in natural language and let an LLM automatically score the results — dramatically lowering the barrier to writing evaluation logic for complex, open-ended questions. It's also one of the mainstream solutions in the industry today for large-scale automated evaluation scenarios.

Final Thoughts

The most tedious yet most valuable part of this topic is precisely these directional, conceptual foundations. The creator particularly reminds readers: the subsequent tool operations and YAML configuration are execution-level details that you can grasp at a glance. What truly determines whether you can hold your ground on a project or go deep in an interview is your understanding of the underlying logic — why AI needs evaluation, what to evaluate, and what to use for evaluation. When you're actually doing the work and find yourself confused about direction, coming back to this framework will be especially valuable.