Agent Evaluation Isn't Just About Building a Framework: Langfuse and a Two-Layer Evaluation Architecture in Practice

Many teams pour all their energy into orchestration and tool calling in Agent projects, only to discover the truly thorny problem after launch: how do you know if it's answering correctly? Based on a real-world financial analysis Agent project, this article lays out three core insights about Agent evaluation, and combines the Langfuse platform with four evaluation frameworks to break down a practical, deployable two-layer evaluation architecture.

Three Insights: Agent Evaluation Is Not Traditional Software Testing

Before diving into Agent evaluation, it's worth clarifying three premises that are easily confused.

First, the goal of Agent evaluation is not 100% accuracy. Because the underlying layer of an Agent is a large model, and whether it's DeepSeek^Ad, OpenAI, or Anthropic, they are all fundamentally probabilistic models—predicting the next token based on the previous one. Attention mechanisms allow the model to understand semantics and logic, but the probabilistic nature makes absolute accuracy very difficult to achieve. The real goal of evaluation is to make the Agent's conclusions and operation results approach 100%, just as 4K gold at 99.99% is already extremely high purity—what engineering pursues is that process of "approaching," not a hallucinatory perfection.

Second, Agent evaluation is not the same as large model evaluation. Large model evaluation typically involves giving a question and looking at the generated answer—relatively straightforward. But an Agent involves multi-step reasoning, tool calling, and state transitions. The object of evaluation shifts from "a single output" to "the entire execution trajectory," which is a completely different level of complexity.

Third, Agent evaluation is not the same as software testing. This is the most common misconception. Traditional software testing verifies whether functionality runs correctly and whether processes flow through; Agent evaluation, on the other hand, must answer "is the answer trustworthy," "is the reasoning sound," and "was the tool selection correct." The former is a deterministic judgment; the latter is a probabilistic one.

These three insights determine the direction of subsequent architecture design: what you need is not just pass/fail, but an evaluation system that is traceable, quantifiable, and iterable.

Project Background: A Financial Analysis Agent on 9,300 Financial Report Records

Discussing evaluation methods requires a concrete scenario. Here we use a financial analysis Agent built on the Harry architecture and Langfuse, positioned as an enterprise-grade evaluation platform.

At the data level, the project collected over 9,300 public financial reports from global listed companies, stored in a lightweight SQLite database in the financial report table. Each record includes the company name, year, PDF file, and text content and tables extracted from the report. This data comes from public sources, not model-generated.

The Agent's task is: based on these public financial reports, answer finance-, fund-, and stock-related analysis questions. For example, "What is the revenue trend for a certain company in a certain year?" or "Is a certain company's cash flow situation healthy?"

The focus of this project is not the Agent's functionality itself, but rather: after the Agent answers a question, how do you determine whether the answer is accurate? This is exactly the problem the evaluation system must solve.

Langfuse: The Infrastructure for Agent Observability

The first step in Agent evaluation is "being able to see." In this project, Langfuse serves as the observability layer—tracing the complete chain of every Agent call.

Specifically, what Langfuse records includes: the input and output of each request, the intermediate state of each reasoning step, the parameters and return results of tool calls, and the overall latency and token consumption. This trace data is the raw material for subsequent evaluation.

Without observability, evaluation has no starting point. You cannot make meaningful judgments about an Agent whose execution process you cannot see. The value of Langfuse is turning the Agent's "black box" into a "gray box"—you don't need to understand the activation values of every neural network layer, but you do need to know which steps it took, which tools it called, and at which step it deviated.

Trace, Span, and Generation are key to understanding Langfuse's data model. A complete Agent call corresponds to a trace, representing the entire request lifecycle from user question to final response. Within a trace are several spans, each corresponding to an execution step in the chain, such as a retrieval, a tool call, or a round of reasoning. When a step involves large model generation, it is recorded separately as a generation, with prompt, completion, token count, and duration. This hierarchical structure lets evaluators both see overall latency and success/failure from the top down, and drill down into a specific span to check whether tool parameters are correct and whether retrieved fragments are relevant. It is precisely this layered recording mechanism that allows component-level evaluation and end-to-end evaluation to share the same trace data, without the need for duplicate instrumentation.

The Two-Layer Evaluation Architecture and Four Evaluation Frameworks

On top of Langfuse's trace data, the project built a two-layer evaluation architecture.

Layer one: Component-level evaluation. This evaluates individual links in the Agent's execution chain, such as whether the retrieved financial report fragments are relevant, whether tool call parameters are correct, and whether single-step reasoning is logically self-consistent. The advantage of this layer is precise localization—when the final answer is wrong, you can quickly identify whether the problem lies in the retrieval step, the reasoning step, or the generation step.

Layer two: End-to-end evaluation. This evaluates the Agent's final output as a whole, judging whether the answer is accurate, complete, and consistent with the source data. This layer is closer to the user's perspective, but its problem-localization capability is weaker than component-level evaluation.

Using both layers together ensures control over the final outcome while providing sufficient diagnostic granularity.

At the evaluation framework level, the project combines four types of evaluation methods: rule-based deterministic checks (such as whether values match the source data), LLM-based judge evaluation (using a large model to judge answer quality), sampling evaluation based on human annotation, and online evaluation based on user feedback. The four cover evaluation needs from different dimensions—rule checks handle hard facts, LLM judges handle semantic quality, human annotation provides baseline calibration, and user feedback reflects real-world performance.

LLM-as-a-Judge is an evaluation paradigm widely adopted in the past two years, referring to using a more capable large model as a judge to score or make judgments on another model's output. Its advantage is that it can handle semantic quality that is hard to describe with rules, such as whether an answer is on topic, whether reasoning is coherent, and whether factual hallucinations exist. A common practice is to provide reference answers or a scoring rubric, letting the judge model score along dimensions. But it also has known biases: it tends to prefer longer answers, is more lenient toward outputs from its own model family, and is position-sensitive (earlier options are more likely to be selected). Therefore, in engineering practice, LLM judges are usually used in combination with rule checks, and human annotation is used for calibration in critical scenarios, to avoid a single judge's bias being amplified into systematic error.

The division of labor between component-level evaluation and end-to-end evaluation essentially corresponds to different granularities in observability. Component-level evaluation targets a single span or generation in the trace, answering local correctness questions, such as whether the retrieved financial report fragment semantically matches the question, and whether the year and company name passed into a tool call are consistent with user intent. Its signal is denser, because a single Agent execution often produces multiple steps, and each step can produce an evaluation result, making it easier to detect regressions quickly when data volume is limited. End-to-end evaluation only looks at the final output; its signal is sparse but closer to business value. A common engineering approach is: use end-to-end evaluation as a go-live gate, and component-level evaluation for problem attribution and iteration guidance—the two are complementary rather than substitutes.

Practical Recommendations

If your team is working on an Agent project, the following points are worth considering:

• Build observability first, then talk about evaluation. Without trace data, evaluation is a castle in the air. Tools like Langfuse should be integrated before launch.
• Accept the probabilistic nature and set reasonable expectations. Don't use 100% accuracy as an acceptance criterion; instead, set quantifiable approaching targets, such as key fact accuracy reaching a certain threshold.
• Run the two-layer architecture in parallel. If you only do end-to-end evaluation, you won't know what to fix when problems arise; if you only do component-level evaluation, you'll easily overlook overall effectiveness. Neither can be missing.
• Cross-validate with multiple frameworks. Any single evaluation method has blind spots—rule checks can't cover semantic quality, LLM judges may have biases, and human annotation is costly but irreplaceable.

Agent evaluation is not a one-time acceptance action, but a continuous process throughout the project lifecycle. Treat the evaluation system as part of the product to be built, not as the final checkpoint before launch.

One-line Summary

Using a financial analysis Agent as an example, this explains how to implement Langfuse observability and a two-layer evaluation architecture.

Paragraph Summary

Centered on a financial analysis Agent built on over 9,300 listed company financial reports, this article lays out three core insights about Agent evaluation: the goal is to approach rather than achieve 100%, Agent evaluation differs from large model evaluation, and it also differs from traditional software testing. The author proposes using Langfuse as observability infrastructure to record the complete call chain, and building a two-layer evaluation architecture on top of it—component-level evaluation for precise localization of problems in retrieval, reasoning, and generation, and end-to-end evaluation for controlling final answer quality. It also combines four frameworks—rule checks, LLM judges, human annotation, and user feedback—for cross-validation. The article concludes with practical recommendations such as building observability first, accepting the probabilistic nature, running both layers in parallel, and cross-validating with multiple frameworks.