The Three-Layer AI Evaluation Framework: LLMs, RAG Knowledge Bases, and Agents

Three Testing Paradigms for the AI Era

Six months ago, the conversation around AI and software testing was still in its infancy — mostly centered on "AI-assisted testing." Today, with companies like Tencent, Alibaba, ByteDance, and DeepSeek pushing hard on engineering-level AI adoption, the path forward for AI testing has become much clearer.

Based on insights from technical discussions in the community, the testing landscape in the AI era can be categorized into three paradigms:

• Testing with AI: Using AI to assist in testing and improve efficiency. This was a hot topic six months ago, but it's largely become old news.
• Testing alongside AI: Humans and AI each playing distinct roles. Humans are no longer the sole decision-makers — requirements analysis, test case design, system exploration, code writing, and automated regression can all be handed off to AI. This is the "AI-led testing" model that leading companies are now embracing.
• Testing AI itself: An entirely new direction — not about dividing work between humans and AI, but about how to test software systems that have AI capabilities built into them.

This third paradigm is rising fast. AI-powered customer service from China Mobile and China Telecom, AI fortune-telling apps, AI video generation tools — more and more products now have AI functionality embedded inside them. Testing these AI-powered applications requires a completely different approach from traditional software testing, and it's becoming an important discipline in its own right.

Why "Testing" Needs to Become "Evaluation"

Testing AI is fundamentally different from traditional software testing. Traditional testing relies on assertions — a result is either right or wrong, pass or fail, with no middle ground. In API testing, for example, you can assert on a status code or assert that a specific field returns a specific value. The outputs are deterministic and reliable.

But large language models break this determinism. Their outputs are probabilistic, driven by two objective factors:

At the Architecture Level: The Inherent Probabilism of Transformers

Most mainstream generative models today are built on the Transformer architecture, and Transformer output is fundamentally a probabilistic token selection process. This means LLMs are inherently non-deterministic — the same question can yield different answers, and that's a core characteristic of how they work.

At the Output Level: Diversity of Expression

Even when a result is conceptually fixed (say, the number "2"), it can be expressed in countless ways: Chinese characters, Roman numerals, Arabic digits, binary, hexadecimal, and so on — not to mention variations in tone and style. Any small change to model parameters, prompts, or context can produce dramatically different outputs.

This means: for a correct scenario, a simple equality assertion might work. But for an incorrect scenario, there could be tens of millions of ways the output could be "not equal." Using assertions to judge AI output simply doesn't scale.

This is why we introduce a new term: evaluation. This isn't a domestic invention — it's the internationally accepted standard for AI testing across academia and industry. Just as we don't judge a person as simply "good" or "bad," and medical diagnoses use scoring rather than binary conclusions, AI — which mimics human behavior — should be assessed through scoring and evaluation rather than pass/fail assertions.

Evaluation Doesn't Mean Abandoning Testing

There's a common misconception worth clearing up: AI evaluation doesn't fully replace traditional testing.

The core of evaluation is scoring — expressing how good or bad a result is using a continuous value (typically 0 to 1). But an AI product ultimately needs a clear verdict: can this software go to production? That's where testing comes in — setting a passing threshold for the scores, much like a college entrance exam cutoff or a clinical diagnostic threshold.

Interestingly, evaluation and testing can convert into each other. In AI, scoring typically uses continuous values between 0 and 1, but you can simplify that to just 0 and 1 — fail and pass — which reduces it back to a traditional assertion-style test. In practice, you'll encounter both continuous scores and binary judgments, so understanding their relationship is key.

Version Pinning: A Special Requirement for AI Evaluation

Because evaluation results can be ambiguous and variable, AI testing demands stricter version management — specifically, two versions must be pinned simultaneously:

• The version under test: Not just the software version number, but also which model was used, what parameters were passed, what prompts were used, and what context was provided. Any change to any of these constitutes a version change. Switching from DeepSeek to GPT, for example, means the results must be re-evaluated from scratch.
• The evaluation version: The means of testing itself — the dataset, assertion rules, scoring criteria, and passing threshold. Any change to the dataset or threshold will also produce different results.

Only by holding one version constant and varying the other can you draw meaningful comparisons (e.g., comparing model quality, or the effect of different thresholds). If both change simultaneously, you lose comparability, and the result must be treated as an entirely independent evaluation. This logic closely parallels performance testing — which also isn't purely about right or wrong, but rather about identifying issues through comparison under controlled conditions.

What to Evaluate: A Three-Layer Progressive Framework

Many people jump straight to metrics — faithfulness, recall, concurrency... but that's putting the cart before the horse. The right approach is to identify the evaluation target first, because different targets have entirely different objectives and metrics.

Evaluation targets fall into three layers, each building on the previous:

Layer 1: Large Language Model (LLM) Evaluation

This focuses on assessing the correctness, completeness, safety, and stability of model-generated content. For model providers like DeepSeek, this is core work. For typical companies, it applies in these scenarios:

• Model selection: Which model is better, faster, or smarter?
• Prompt optimization: Iterating based on evaluation results, not gut feeling
• Fine-tuned model validation: Open-source small models like Qwen can now be fine-tuned locally, and many companies have real verification needs here

Layer 2: Knowledge Base (RAG) Evaluation

RAG systems work by first retrieving relevant documents, then using them to answer user questions. Evaluation needs to cover every step:

• Is retrieval accurate?
• Is recall complete?
• What is the faithfulness — does the generated response actually stay grounded in the specified reference material?

Metrics like recall and faithfulness exist specifically for RAG evaluation scenarios.

Layer 3: Agent Evaluation

Customer service agents, data agents, and vertical-domain agents in travel, legal, and healthcare will only become more prevalent. Agent evaluation is the most complex, because agents involve task planning, tool selection, task orchestration, knowledge base integration — and at the foundation, a large language model.

These three layers aren't a fork in the road where you pick one path — they represent a progressively increasing level of knowledge and complexity: start with LLM evaluation, move on to RAG evaluation, and finally tackle agent evaluation. This also forms a clear learning roadmap for AI testing.

Tool Selection: Why Promptfoo Is Worth Considering

The final key question is: what tool should you use for AI evaluation? Among the code-friendly evaluation tools available, Promptfoo stands out for several reasons:

• CLI + YAML-driven: It can be invoked by AI itself — a tool for evaluating AI that can also be used by AI. This creates enormous potential for "AI evaluating AI"
• Human evaluation support: In a world where AI isn't yet the definitive authority, human evaluation remains the gold standard. Test case assertions written in YAML enable a zero-code evaluation workflow
• Strong extensibility: Supports batch calls and Python customization, with a promising development trajectory
• Acquired by OpenAI: Its quality has been validated by one of the world's top AI companies

Of course, tool selection should always match the use case: academic model benchmarking and production monitoring for live agents each have more suitable tools. But for most testing teams whose primary goal is determining whether software is ready to ship, Promptfoo's acceptance testing capabilities — combined with coverage across LLM, RAG, and agent evaluation — make it a well-rounded choice.

Summary

The shift from assertions to evaluation is a conceptual transformation that every test engineer must make in the AI era. Understanding why we evaluate, what we evaluate, and how we evaluate matters far more than mastering any specific tool.

When you feel lost navigating AI testing in your day-to-day work, come back to this three-layer progressive evaluation framework — from LLMs to knowledge bases to agents, building depth step by step. That's where the real value lies.