Why Is Agent Evaluation Harder Than RAG? A Hands-On Breakdown from Financial Analysis

Agent evaluation is becoming the most underrated part of deploying large models in production. A teacher at Mashi Education who specializes in large models hosted a Bilibili livestream, using the 12th project currently being taught—an agent evaluation platform built on Langfuse—as the backdrop to systematically break down the fundamental differences between agent evaluation, RAG evaluation, and traditional software testing. This sharing session was not vague methodology, but rather built on a real running financial analysis agent: it collected over 9,300 financial reports from global publicly listed companies, stored them in a lightweight SQLite database, and the agent answers finance, fund, and stock analysis questions based on this public financial data.

Three Key Insights About Agent Evaluation

Insight One: Probabilistic Models Mean 100% Accuracy Does Not Exist

The foundation of an agent is a large model, and whether it's DeepSeek^Ad, OpenAI, or Anthropic, these models are fundamentally probabilistic—they compute the probability of the next token based on the previous one. Although attention mechanisms allow the model to understand semantics and logic, the probabilistic nature means it struggles to achieve 100% accuracy.

The goal of evaluation is to make the agent's conclusions or operational results (approach) 100%, not (reach) 100%. The instructor repeatedly emphasized in the project: no matter how many layers of evaluation you build, an agent can roughly achieve 99.9%, but can never truly reach 100%. This insight directly determines the design philosophy of the evaluation system—not pursuing absolute correctness, but establishing a tracking mechanism that continuously approaches the upper limit.

What Are Tokens and Autoregressive Generation

To understand why probabilistic models cannot achieve 100% accuracy, you first need to understand tokens and the autoregressive generation mechanism. A token is the smallest unit of text a model processes—it could be a word, a subword, or even a character. Autoregression means the model predicts only the next token each time, then appends the predicted result to the end of the input sequence, and predicts the next token based on the extended sequence, looping until an end-of-sequence token is generated. Each prediction step comes from a probability distribution, from which the model samples or takes the highest-probability token. This means that even with identical input, under sampling strategies with a temperature parameter greater than 0, multiple runs may produce different token sequences, leading to different tool-calling paths and reasoning processes. This inherent randomness is the fundamental source of why probabilistic models cannot guarantee 100% determinism, and it is also why evaluation systems must be designed around distributions and trends rather than single-point judgments.

Insight Two: Agent Evaluation ≠ Large Model Evaluation

The logic of large model evaluation is: pose a question, let the model generate an answer, and evaluate the model's capability based on the answer. This process is relatively straightforward.

Agent evaluation is completely different. An agent not only has a final response, but also a tool-calling chain, a data-reading process, and intermediate reasoning steps. The evaluation target shifts from a single output to the entire execution chain, and the complexity naturally rises by an order of magnitude.

What Are Tool Calling and Execution Chains

The core capability that distinguishes an agent from an ordinary large model is tool calling (Tool Calling). During reasoning, an agent can decide to call external functions or APIs—for example, executing SQL queries, reading files, or performing numerical calculations—and then use the results returned by the tools as new context for continued reasoning. A complete agent execution forms a chain containing multiple nodes: user query, intent understanding, tool selection, parameter generation, tool execution, reading returns, secondary reasoning, and answer generation. Every node on this chain can fail—for example, selecting the wrong tool, filling in parameters incorrectly, or filtering the time range improperly. Evaluating a large model only requires examining the final text output, but to evaluate an agent, you must reconstruct and inspect the entire execution chain. This is the direct reason why the complexity differs by an order of magnitude.

Insight Three: Agent Evaluation ≠ Traditional Software Testing

Many people mistakenly believe that agent evaluation is just testing whether software runs properly. This analogy is fundamentally wrong.

Traditional software testing focuses on whether functionality executes as expected, and the results are deterministic. Agent evaluation deals with probabilistic outputs—the same input may produce different tool-calling paths and reasoning processes. Using a deterministic framework to evaluate a probabilistic system is itself a mismatch.

Where the Difficulty Lies in Evaluating a Financial Analysis Agent

One Example Explains the Problem

Suppose you ask the agent: "For Xinhe Manufacturing, how much did annual revenue grow in year five compared to year four?" The agent answers "20%."

The traditional approach is to directly check whether the answer is correct—if the standard answer is 20%, it's judged correct. But this judgment method has two fatal flaws:

In the first case, the agent happened to calculate incorrectly, but the result coincidentally equals 20%. The answer is right, but the entire process is wrong.

In the second case, the context happens to mention that a company's revenue grew by 20%, but that was "year three" growth, not "year five." The large model failed to identify the year difference and directly used data from the wrong range as the answer. The answer looks correct, but the data source is completely wrong.

Judging correctness solely by the final answer cannot distinguish between "truly understanding and calculating correctly" and "coincidentally guessing right."

Evaluation Must Cover the Complete Chain

Agent evaluation must satisfy three conditions simultaneously to judge an answer correct:

• Reasonable result: The final output value or conclusion matches expectations
• Reasonable evidence: The tool called the correct data source, and the data range read, time filter conditions, and field selections are all correct
• Reasonable tool-calling chain: From data query to formula calculation, the numerator and denominator are correctly chosen, and the entire reasoning chain is unbroken

All three conditions are indispensable. This explains why agent evaluation is more complex than RAG evaluation—RAG evaluation mainly focuses on retrieval quality and generation quality, while agent evaluation needs to trace tool calls, data flow, and the complete chain of multi-step reasoning.

What Is RAG Evaluation

RAG (Retrieval-Augmented Generation) evaluation typically focuses on two core aspects: retrieval quality and generation quality. Retrieval quality concerns whether recalled documents are relevant and whether ranking is reasonable, commonly measured by metrics such as hit rate, recall, and MRR; generation quality concerns whether the answer is faithful to the retrieved context and whether it answers the user's question, commonly measured by metrics such as faithfulness and answer relevance. In comparison, RAG's chain is shorter—basically three steps: "retrieve—concatenate—generate," with no intermediate tool parameter construction or external execution steps. Agent evaluation, beyond these three steps, must also trace whether tool selection is correct, whether parameter passing is accurate, and whether data ranges and filter conditions match. Therefore, it has more evaluation dimensions, a longer chain, and higher requirements for tracking infrastructure.

Langfuse Evaluation Platform in Practice: From Data to Tracing

This project uses Langfuse to build an agent evaluation platform. At the data level, it collected over 9,300 financial reports from global publicly listed companies, stored in a SQLite database. The table is named financial_report, containing company name, year, PDF file path, text content, and table data.

This data comes from public sources—real financial reports published by publicly listed companies. The agent answers financial analysis questions based on this data, and the evaluation system needs to trace the complete chain behind each answer.

The instructor mentioned that while teaching this project, he found the evaluation phase "quite interesting," so he dedicated time to discuss it in the livestream. The project is nearing completion, with about three hours of content remaining.

What Are Langfuse and Chain Tracing

Langfuse is an open-source observability and evaluation platform for LLM applications, designed specifically for large models and agent applications. It provides core concepts such as trace, span, and generation: a complete agent request corresponds to one trace, and each step in the chain (such as a tool call or a model generation) corresponds to a span or generation. Developers can use SDK instrumentation to automatically record the input/output, latency, token consumption, and parameter information of each step. Based on these structured logs, Langfuse supports defining evaluation functions, scoring, comparing experimental versions, and visualizing evaluation results. In this project, its role is precisely to turn the agent's complete execution chain into queryable, traceable records, thereby providing a data foundation for the judgments of "reasonable evidence" and "reasonable chain," rather than relying solely on final answer string comparison.

Engineering Implications of Agent Evaluation

Evaluation is not an additional step after agent development is complete, but rather infrastructure that runs throughout the entire development process. Several engineering practice directions can be extracted from the instructor's sharing:

The evaluation target shifts from results to chains. You need to record every step of the agent's tool calls, parameter passing, data reading, and intermediate reasoning to judge whether the final answer is truly reliable.

Accept the probabilistic nature. Evaluation metric design should revolve around "approaching" rather than "equaling," focusing on distributions and trends rather than single-point judgments.

Distinguish evaluation layers. Large model evaluation, RAG evaluation, and agent evaluation are three different levels of problems and cannot be mixed with the same methodology.

The complexity of agent evaluation stems from the complexity of the system itself. When an agent evolves from a simple Q&A tool into an autonomous system capable of calling tools, reading data, and executing multi-step reasoning, evaluation must also upgrade from answer comparison to chain tracing. This is not a process that can be simplified, but a threshold that agents must cross to enter production environments.

One-line Summary

Agent evaluation is harder than RAG due to probabilistic nature and multi-step chains, requiring tracing of results, evidence, and calling chains.

Paragraph Summary

Based on a real-world financial analysis agent project, this article systematically breaks down why agent evaluation is harder than RAG evaluation and traditional software testing. The agent's foundation is a probabilistic model, so the evaluation goal can only be approaching 100% rather than reaching 100%, with 99.9% already being the upper limit. Unlike large model evaluation, which only examines final output, agent evaluation must trace the complete chain of tool calls, data reading, and multi-step reasoning. Through the case of "20% revenue growth," the article shows that judging correctness solely by the final answer misses situations like wrong calculations coincidentally matching or incorrect data sources. Three conditions must be met simultaneously: reasonable result, reasonable evidence, and reasonable tool-calling chain. The project collected over 9,300 global listed company financial reports into SQLite, built an evaluation platform on Langfuse, and upgraded evaluation from result comparison to chain tracing.