AI Agent Evaluation in Practice: Tool Calling, Trajectory Matching, and LLM Judges

Title

AI Agent Evaluation in Practice: Tool Calling, Trajectory Matching, and LLM Judges

Meta Description

AI Agent evaluation needs to be layered: first use deterministic checks to verify tool calls, then use trajectory matching to ensure the execution path is reasonable, and finally use LLM judges to assess response quality. This article combines practices with Pydantic Evals and Mastra to explain the design and implementation of a three-layer evaluation framework.

Article Content

Building AI Agents is only the first step—how to evaluate them reliably is the real challenge. Unlike traditional LLM output evaluation, Agent behavior is sequential: it calls tools, observes results, and decides what to do next. Simply comparing final answers ignores critical failures in the process. This article introduces a layered evaluation framework: first perform deterministic tool-call checks, then trajectory matching, and finally use LLM judges for open-ended quality assessment.

Why Agent Evaluation Needs to Be Layered

At its core, an Agent is a loop: think → call tools → observe → think again. The final output may be correct, but the process may be full of ineffective calls, repeated attempts, or dangerous intermediate steps. For example, a customer service Agent may ultimately give the correct answer, but before that it mistakenly called the refund API three times. If only the final output is evaluated, this flaw will be completely hidden.

Therefore, effective Agent evaluation must cover three levels simultaneously: whether tool calls are correct, whether the execution trajectory is reasonable, and whether the final response quality meets the standard. These three levels range from deterministic to fuzzy and correspond to different evaluation techniques. The core argument of the original article is precisely this: evaluate in layers according to the degree of determinism, prioritize cheap and reliable checks, and introduce LLM judges only when necessary.

Layer 1: Deterministic Tool-Call Checks

This is the most basic and fastest evaluation layer. Every tool call made by an Agent has an explicit name and parameters, which can be asserted like unit tests. For example, check whether search_flights was called, whether the date format in the parameters is correct, and whether available seats were checked before booking.

These checks are fully deterministic, do not require an LLM, and have extremely low running costs. They can catch a common class of errors: the Agent bypassed a key tool, passed incorrect parameters, or called a tool that does not exist. In continuous integration, this layer should be a mandatory run for every commit. Frameworks such as Pydantic Evals provide assertion primitives for tool calls, and Mastra also has similar built-in check capabilities in its evaluation module.

Pydantic Evals is an evaluation framework launched by the Pydantic team, designed specifically for LLM and Agent applications. It leverages Pydantic's data validation capabilities to define evaluation cases as strongly typed models, thereby catching type errors at compile time. For tool-call checks, Pydantic Evals allows developers to declare expected tool names and parameter schemas and automatically verify whether actual calls match. Its core advantage is seamless integration with the Python type system, making evaluation logic both safe and easy to maintain. The framework also supports asynchronous evaluation and batch runs, making it suitable for integration into CI/CD pipelines.

Mastra is an Agent development framework for TypeScript with a built-in evaluation module. It is deeply coupled with the Agent runtime and can directly intercept and record detailed information about each tool call, including input parameters, returned results, and latency. Mastra's evaluation functionality supports declarative assertions, allowing developers to write check rules for tool-call sequences and easily integrate them into existing testing systems. Because it comes from the same source as the runtime, Mastra has a natural advantage in capturing intermediate states, making it particularly suitable for complex Agent scenarios that require fine-grained debugging.

Layer 2: Trajectory Matching

Correct tool calls do not mean the order is reasonable. Trajectory matching focuses on the Agent's execution path: does it operate in the expected sequence of steps? Are there unnecessary loops? Did it waste turns on the wrong branch?

There are usually two approaches to trajectory matching. Exact matching is suitable for scenarios with fixed processes, such as "check inventory → lock inventory → place order," where any deviation is considered a failure. Subsequence matching is more lenient: it only requires key steps to appear in order and allows exploratory calls in between. In real projects, subsequence matching is more practical because it tolerates reasonable Agent exploration while ensuring that critical safety checks are not skipped.

Trajectory data is also a goldmine for debugging. When evaluation fails, the complete trajectory log can directly point out where the Agent deviated from expectations, which is much more efficient than looking only at the final answer.

Layer 3: LLM Judge

The first two layers cover "whether it did it right," but they cannot judge "whether it did it well." The tone and completeness of the final response, and whether it is based on the real data returned by tools—these require semantic understanding. LLM judges are designed for this: use a stronger model to score the Agent's final output according to grading criteria.

The key to an LLM judge lies in the design of the grading criteria. Vague instructions (such as "evaluate answer quality") will produce unstable scores. Effective judge prompts need clear dimensions: factual accuracy (whether it is consistent with tool returns), completeness (whether it covers all user questions), and safety (whether sensitive information is leaked). At the same time, a small number of annotated examples should be provided, and the judge should output structured scores and reasons for manual auditing.

It must be emphasized that an LLM judge should not be the only method. It is costly, stochastic, and may be deceived by the style of the Agent's output. Placing it in the last layer of the layered framework and running it only after the first two layers pass is the most economical approach.

Working Code and Framework Selection

The original article provides runnable code examples for Pydantic Evals and Mastra. The advantage of Pydantic Evals lies in its deep integration with the Python type system, allowing assertions for tool calls to be written as type-safe expressions; Mastra is better suited to the TypeScript stack, and its evaluation module is tightly coupled with the Agent runtime, enabling it to directly capture runtime tool-call events.

When choosing a framework, the core consideration is the depth of integration between the evaluation logic and the Agent runtime. If evaluation needs access to the complete intermediate state (such as the raw return of each tool call), choosing a framework from the same source as the runtime will be easier. If only black-box evaluation is needed, any tool that can parse trajectory logs will do.

Practical Recommendations

Start with deterministic checks. First write assertions for all key tool calls; this can catch most low-level errors at almost zero cost. Then add trajectory matching for core processes, focusing on safety checks and irreversible operations. Finally, introduce LLM judges for user-facing responses and regularly manually spot-check the consistency of the judges themselves.

Evaluation is not a one-time task. As Agent capabilities expand, the evaluation set needs to grow accordingly. Only by turning every failure case discovered online into a new evaluation case can the evaluation system truly keep up with the evolution of Agents.

One-line Summary

AI Agent evaluation needs to be layered: tool-call checks, trajectory matching, and LLM judges.

Paragraph Summary

This article proposes a layered AI Agent evaluation framework to address the problem that traditional final-output evaluation ignores process defects. The framework includes three levels: the first is deterministic tool-call checks, which catch low-level errors by asserting tool names and parameters at extremely low cost; the second is trajectory matching, which focuses on execution order and loops, with subsequence matching being more practical than exact matching; the third is LLM judges, which assess the semantic quality of the final response but require clear grading dimensions and examples. The article also introduces frameworks such as Pydantic Evals and Mastra and recommends starting with deterministic checks, gradually introducing trajectory matching and LLM judges, and turning online failure cases into evaluation cases to maintain the effectiveness of the evaluation system.