A Practical Guide to AI Model Evaluation: Transitioning from Assertions to Assessment-Based Testing

AI is reshaping both development and testing simultaneously. Developers now need to learn testing, while test engineers need to learn how to write code. As one Bilibili creator who transitioned from QA to development — now building projects in JavaScript — puts it: "The industry is changing, and learning can't stop." This article, based on their AI evaluation masterclass, systematically breaks down the evolution of testing in the AI era, the fundamental shift in core concepts, and practical evaluation paths for LLMs, RAG systems, and Agents.

The Three-Layer AI Testing Framework: From Assistance to Leadership

What was a vague sense of direction six months ago has now crystallized into a much clearer industry landscape. The author introduced three AI testing paradigms early in their public courses — and all three are now becoming reality.

The first is testing with AI — using AI as a tool to assist testing and boost efficiency. Six months ago this was the hottest topic in QA circles, but it's rapidly becoming table stakes.

The second is testing alongside AI — AI-led testing. Companies like Tencent, Alibaba, and ByteDance, which are pushing hardest on AI in China, along with DeepSeek's growing focus on engineering research, are all driving this model. In this paradigm, humans are no longer fully in control. Requirements analysis, test case design, system exploration, code writing, and automated regression can all be handed off to AI, with humans and AI working in complementary roles.

The third is testing AI itself — or AI application testing. As more software products integrate AI features — AI customer service, AI fortune-telling, AI video generation, and beyond — testing these AI-powered applications has become an entirely new challenge. The thinking and process here are fundamentally different from traditional software testing, and it's the central focus of this article.

Why "Evaluation" and Not "Testing": The Gap Between Determinism and Probability

The cornerstone of traditional software testing is the assertion — right or wrong, pass or fail, with no third option. You assert a status code, a response field, an equality condition. Output is consistent, and assertions are reliable.

But generative AI shatters this determinism. The probabilistic nature of LLMs can be understood from two angles:

Architectural Probability at the Model Level

Most mainstream generative models today are built on the Transformer architecture, which has probability-based selection baked into its output mechanism. Ask an AI the same question and you may get different answers each time — this probabilistic nature is hardwired into the architecture itself.

The probabilistic output of the Transformer architecture stems from its autoregressive generation mechanism: the model predicts only one token at a time, producing a probability distribution over the entire vocabulary. A sampling strategy (such as Top-K or Top-P) then randomly draws from this distribution rather than always selecting the highest-probability token. The temperature parameter directly controls how "flat" this distribution is: higher temperatures give lower-probability tokens a greater chance of being selected, producing more varied and creative outputs; as temperature approaches zero, the model almost always picks the most probable token, yielding more stable but potentially repetitive results. This is why the same question asked at different moments can produce entirely different answers — by design, the model is not a deterministic function.

Output Diversity at the Result Level

Even when the underlying answer is semantically fixed, the surface form can vary enormously. Take the number "2" — it can appear as Chinese characters (traditional or simplified), Arabic numerals, binary, hexadecimal, or ASCII. Style can vary just as wildly. Add in model parameters, temperature settings, and other subtle changes, and the results can diverge significantly.

For all these reasons, simple equality assertions no longer work. The author offers an intuitive analogy: judging a person as simply "good" or "bad" has obvious limitations — just as college entrance exams use numerical scores rather than a binary "eligible/not eligible" to capture meaningful distinctions, and medicine uses scoring scales for IQ, cancer staging, and depression severity. AI, which is meant to mimic human behavior, should likewise be measured through scoring and evaluation.

Evaluation Doesn't Replace Testing: You Still Need a Passing Threshold

A common misconception needs clearing up: evaluation doesn't entirely replace testing. The author emphasizes that "evaluation" refers to the act of scoring, but after scoring, a testing step is still required — setting a passing threshold for those scores.

Just as college admissions have cutoff scores and oncology diagnoses have scoring thresholds, the decision of whether an AI product is ready to ship ultimately requires a clear conclusion — one derived by comparing evaluation scores against a defined threshold.

In AI evaluation, scores are typically expressed as continuous values between 0 and 1 (e.g., 0.1, 0.3, 0.999). But they can also be discretized: 0 for fail, 1 for full marks, which reduces back to traditional assertion-style testing. So evaluation can be continuous or binary — these two approaches aren't mutually exclusive.

Version Pinning: The Dual-Version Principle in AI Evaluation

Because AI evaluation is complex, ambiguous, and potentially volatile, the author places particular emphasis on version pinning — and specifically, pinning two versions simultaneously:

The version under test — i.e., the system being evaluated, including which model is used, what parameters are passed, what prompts are applied, and what context is added. Any change to any of these means a new version under test. Switch from DeepSeek to GPT mid-evaluation, and your results are no longer comparable — they need to be interpreted as a fresh set of data.

The evaluation version — i.e., the methodology used to test, including the dataset, assertion rules, evaluation prompts, and threshold values. Change the dataset or threshold, and the results change too.

The key principle: meaningful comparison is only valid when exactly one version changes at a time. Use the same evaluation methodology with different models to compare model quality; use the same system under test with different thresholds to compare threshold effects. If both versions change simultaneously, the comparison loses its validity — you're looking at two independent results that must each be interpreted on their own terms.

The author also draws a practical analogy: AI evaluation closely resembles performance testing. Both care less about individual pass/fail outcomes; both surface issues by running a dataset under specific conditions; and both require at least two comparable runs before any meaningful conclusion can be submitted. Engineers with performance testing experience will find this framework especially intuitive.

What to Evaluate: Three Progressive Layers — LLM, RAG, and Agent

Many people jump straight to debating metrics like "faithfulness" or "recall" — but the author argues this is the wrong starting point. You must first identify the evaluation target, because different targets have completely different goals and metrics. There are three layers, each building on the previous:

Layer 1: Large Language Model (LLM) Evaluation

This layer primarily assesses the correctness, completeness, safety, and stability of model-generated content. For model companies like DeepSeek, this is a core requirement. For most other companies, LLM evaluation serves three purposes: model selection (which model is better, faster, or smarter), prompt optimization (using data to decide which prompt performs better rather than guessing), and post-fine-tuning validation (quantifying the effect of fine-tuning). With models like Qwen now releasing compact, fine-tunable versions that run on personal computers, post-fine-tuning evaluation has become increasingly relevant.

Layer 2: Knowledge Base (RAG) Evaluation

RAG (Retrieval-Augmented Generation) retrieves relevant documents first, then generates answers based on them. Evaluation must therefore cover every stage of the pipeline: whether retrieval is accurate, whether recall is complete, and whether the generated output is correct. Recall is typically used in knowledge base scenarios to measure whether relevant documents were fully retrieved. Faithfulness measures whether the generated output is grounded in the retrieved sources — whether the model is answering based on the provided documents rather than "improvising."

The core idea of RAG (Retrieval-Augmented Generation) is to combine an external knowledge base with a language model: first, use vector retrieval or keyword matching to pull relevant passages from a document store, then inject those passages as context into the prompt so the model can generate answers with documentary backing — reducing hallucinations and improving timeliness. When evaluating a RAG system, Recall measures "did we retrieve all the relevant documents?", Precision measures "are the retrieved documents actually relevant?", and Faithfulness specifically checks how much of the model's response can be traced back to the retrieved documents. Each dimension maps to a distinct failure mode in retrieval and generation — none can be omitted.

Layer 3: Agent Evaluation

Customer service agents, data agents, and domain-specific agents in travel, legal, and healthcare applications are becoming increasingly prevalent. Agents involve multiple steps — task planning, tool calling, and task execution — so their evaluation metrics differ again. This is the most complex layer, as agents typically incorporate a knowledge base internally and rely on a language model to reason, making them a synthesis of the two preceding layers.

These three layers aren't alternatives — they're progressive. Master LLM evaluation first, then move to RAG, and finally tackle the most complex layer: Agent evaluation.

An Agent is a system built on top of a large language model that introduces a "perceive–plan–act" loop: the model doesn't just generate text — it can invoke external tools (web search, database queries, code execution, API calls, etc.) and continue reasoning based on tool outputs, iterating across multiple turns until a goal is achieved. This makes evaluation far more complex than single-turn Q&A. Beyond the quality of the final answer, you must also assess whether task decomposition is reasonable, whether the tool-calling sequence is correct, whether intermediate steps contain redundancy or dead ends, and how errors propagate and amplify across long chains. Agent evaluation currently lacks mature standards; Trajectory Evaluation — scoring the full tool-calling path rather than just the final output — is one of the leading research directions.

Tool Selection: Why Promptfoo Is the Recommended Choice

Once you've defined your evaluation framework and targets, the final step is choosing the right tooling. The author reviews several options and recommends Promptfoo as the standout choice, for the following reasons:

First, it's a CLI + YAML-driven tool. The CLI interface means it can be invoked by AI itself — a tool for evaluating AI that AI can also use, making the "AI evaluates AI" loop genuinely viable.

Second, it supports human evaluation (writing YAML test cases with assertions — human evaluation remains the gold standard in AI testing), batch execution, and custom extensions in Python, making it highly flexible.

Most importantly, Promptfoo has been acquired by OpenAI, a strong validation of its quality by the world's leading AI company. It also covers all three evaluation targets — LLM, RAG, and Agent — addressing the full evaluation lifecycle with a single tool.

The author also offers flexible guidance for other use cases: for academic metric research, a dedicated metrics tool like RAGAS may be more appropriate; for monitoring production agents in live environments, an observability-focused tool makes more sense. But for most test engineers — where evaluation serves a "release readiness" or acceptance-testing function — Promptfoo is the better fit.

Direction Matters More Than Tools: The Core Competency for AI Testing Transitions

The author repeatedly emphasizes that the most tedious yet most valuable content in the course is precisely this theory and direction — the conceptual shift from assertions to evaluation, the dual-version pinning principle, the three-layer progressive evaluation framework, and the logic behind tool selection. The hands-on tool walkthroughs that follow, by comparison, are secondary: "a quick look is enough."

Because when you actually get to work, the problems you encounter aren't usually execution-level issues — they're directional confusion. Once you understand the underlying logic of the shift from testing to evaluation, you can navigate the big picture accurately in interviews or on the job, even without hands-on experience. That's the core competitive advantage for test engineers making the transition in the AI era.