Production-Grade AI Agent Evaluation in Practice: From Tracing to an Automated Closed Loop

Why Does Your Agent Always "Crash" After Launch?

Many teams ship AI features in a nerve-wracking way: build, run a few queries, ask "does this look right?", then go live. The problem is you only tested three times, and three times is not a test suite. When an Agent faces inputs it has never seen, failure is inevitable.

It gets worse: Agents are harder to evaluate than ordinary LLM applications. A single call is "input → output," whereas an Agent makes a series of tool calls and decisions, and every step in the chain can go wrong—you must also judge whether the intermediate steps are correct. Multi-agent systems are even harder: task handoffs, routing, expert processing—each layer adds new ways to fail.

This is why Evals exist: they are the "unit tests" for AI, and Tracing is the "logging" for AI. Tracing tells you what happened; Evals tell you whether it worked well. Without this infrastructure, you are flying by the seat of your pants. Practitioners have summed up a path: ship by feel first, discover it doesn't scale, then start building evals.

The Two Pillars of an Evaluation System: Code Evals and LLM Judges

Code evals are deterministic functions: they don't call a model, don't cost money, and complete instantly. Check whether the output is valid JSON, whether it's under 500 tokens, whether it contains the specified stock ticker—none of this needs an LLM. Its advantages are speed, low cost, and full reproducibility; its drawback is limited flexibility—two differently worded but both correct answers may be misjudged. So you need to adjust your expectations for "unit tests."

LLM-as-judge is far more flexible and can understand semantics: is the answer faithful to the source? Is the tone right? But running it costs money, and it is itself non-deterministic and can make mistakes, so it needs to be calibrated with human judgment.

Human evaluation is the gold standard of quality, but it is slow and expensive. A not-so-fun fact: human annotators, due to fatigue, may miss up to 50% of defects. The three methods are complementary, and most real-world applications adopt a combined strategy.

The selection principle is clear: for deterministic answers (format, length, banned words), use code evals; for semantic understanding (accuracy, faithfulness), use an LLM judge; for failure modes you've never seen, keep a human in the loop, and use human annotations to verify whether the LLM judge is reliable.

Read Traces First, Write Evals Second: Error Analysis Is the Most Underrated Step

Before writing any evaluator, read a dozen or so traces. This is the highest-value activity in the entire workflow, yet it is often skipped by tutorials. Anthropic specifically invested in tools for viewing traces, and teams spend a lot of time every day reading these records—it can distinguish between "the Agent really made a mistake" and "the grader wrongly killed a correct solution."

Reading uses qualitative research methods: open coding, recording each problem you see one by one, describing it in natural language, without categorizing it prematurely. After aggregating, perform axial coding—group similar problems into categories—and then ask about root causes: is it retrieval failure? Reasoning error? Hallucination? Scope violation? Each root cause points to a different fix.

You end up with a table of root-cause frequencies. What to fix first? Decide by multiplying severity by frequency—if there is a one-percent chance of giving dangerous instructions, fix that first.

At the same time, define the criteria for "good" in advance. A good financial report should cite the correct stock ticker, contain real recent data, and distinguish forward-looking analysis from historical summary. These criteria seem obvious when written down, but many teams have never done it and jump straight to responding to user complaints. And the people who often hold a good definition are "non-technical" people like product and customer support—they know best what failure looks like.

Open coding and axial coding originate from Grounded Theory, a qualitative research method. Open coding requires researchers to suspend preconceived categories, read the raw material line by line, and attach a descriptive label to each phenomenon—for example, "the Agent called the wrong API" or "the answer cited an outdated earnings report." At this stage you only record, without categorizing, in order to preserve the original richness of the phenomena. Axial coding then looks for associations among the labels produced by open coding, aggregates similar phenomena into categories, and organizes the relationships among categories around "conditions—actions—consequences" to extract root causes. The benefit of applying this process to Agent error analysis is that it lets failure modes emerge on their own, rather than forcing them into pre-conceived categories, so you can discover failure types the designers never anticipated.

Building and Calibrating Three Types of Evaluators

First line of defense: code evals. The simplest example is checking whether the target stock ticker appears in the report. An easily overlooked principle: evaluate the results the Agent produced, not the path it took. People often want to check the order of tool calls, but Agents frequently find valid paths the designers never anticipated, and judging the path punishes creativity.

Built-in LLM evals. Platforms usually come with templates for accuracy, faithfulness, tool selection, and so on. But pay attention to the application scenario: in an Agent with real-time web search, accuracy evals may fail across the board—the judge model's knowledge cutoff predates the latest data the Agent retrieved, so it complains about "information from the future." In that case, faithfulness evals are more appropriate, because the judge receives the same context as the Agent and can determine whether the report is grounded in the original material. The two evals give different signals—the difference is not about better or worse, but about which question you are actually asking.

Custom evals. Built-in templates cannot cover every use case. A good rubric includes: defining the judge's role and context; explicitly listing specific, observable pass/fail criteria; using XML tags to separate instructions from data; and providing examples of excellent and poor outputs (examples improve consistency the most, but beware of overfitting). Keeping binary options (pass/fail) is usually better than a 1-5 scale, because LLMs are poor at fine-grained distinctions. Requiring the judge to reason before scoring significantly improves accuracy.

Calibrating the judge (meta-evaluation). Treat the LLM judge as a classifier: compare its predictions against a human-labeled golden dataset. In practice, agreement is often only around fifty percent—at that point, read the explanations of the disagreement cases to determine whether the rubric is ambiguous or the human annotation is wrong. Iterate on the rubric, then validate on a held-out set to avoid overfitting to the golden dataset. Eval prompts should also be version-controlled like code.

Common anti-patterns: treating eval prompts as code and casually changing words (they are extremely sensitive to wording); building an "all-in-one judge" that checks every dimension (it cannot be calibrated and failures cannot be localized); and confusing guardrail metrics (hard conditions that must block a release) with north-star metrics (the goals you are pursuing).

Meta-evaluation refers to evaluating the evaluator itself—that is, treating the LLM judge as a binary classifier and measuring its accuracy, recall, and agreement against a human-labeled golden dataset. The most commonly used metric in practice is simple agreement (the proportion of cases where the judge's decision matches the human annotation); a more rigorous approach is Cohen's kappa, which subtracts the baseline of random agreement and reflects true consistency. An agreement rate of only fifty percent means the judge is performing close to a coin flip, and its scores cannot be trusted directly. The diagnostic path is to sample disagreement cases and read the judge's reasoning, distinguishing two types of error: the rubric itself is ambiguous (needs revision) or the human annotation is wrong (needs corrected golden labels). A calibrated judge still needs to be validated on a held-out set to prevent overfitting to the golden set.

From Failure to Fix: Datasets, Experiments, and the Coding-Agent Closed Loop

Filter out all the failure traces and build a failure dataset, then run new prompts against it in a targeted way—faster and cheaper. At the same time, keep a regression dataset to ensure your changes haven't broken what was already working. Once real traffic arrives, replace synthetic data with production failure cases; the union of failures and passes is your golden dataset.

The core abstraction of an experiment is a Python function: take a sample from the dataset, run the task, return the output. A controlled comparison means the same inputs and the same evaluator, with the only variable being the prompt—so score differences can be attributed to your change.

Going further: have a coding Agent pull the failure traces and the judge's explanations, and turn those explanations into concrete changes to the codebase. Every change can be traced back to a specific failure—this is data-driven prompt engineering. Two guardrails are essential: clearly state the goal (otherwise the Agent will cheat to pass the evals, e.g., by incorporating the test data into its own evaluation); and require it to look for common problems rather than chasing individual failure records.

As for sample size: workshop-scale 12-20 samples provide a directional signal; a formal release decision requires about 200-400 samples, still within the range humans can handle. You need to quadruple the sample size to meaningfully reduce the margin of error, so returns diminish clearly.

Controlled experiments in the evaluation context correspond to the A/B testing idea from software engineering: fix the dataset, evaluator, model, and sampling parameters, and change only one variable (usually the prompt), so that score differences can be attributed to that variable. This requires the evaluation pipeline itself to be sufficiently deterministic—the non-determinism of the LLM judge introduces noise, so the same configuration often needs to be run repeatedly and averaged, or the judge's temperature set to zero. As for sample size, 12-20 samples only give a directional signal, and statistical power is insufficient to support release decisions; 200-400 samples narrow the confidence interval to an actionable range. The margin of error only halves when the sample size quadruples—this square-root relationship explains why blindly expanding the dataset has diminishing returns, and why it is better to prioritize improving the quality and representativeness of individual samples.

The Production Loop: Online Evals and Continuous Iteration

Going live is not the finish line. Online evals run automatically on real traffic, can be triggered at the span or trace level, and are sampled by percentage (1%-10%) to control cost. Failure cases are saved as a regression dataset, fed back to the coding Agent, validated through experiments, and then the fix is released—re-entering the loop.

Every failure becomes a new test case, and over time this forms a database unique to your application—tailored to your failure modes, your users, and your quality standards. No one else has this data, and it grows every time the Agent runs—this is precisely your competitive advantage.

In terms of priorities, fixing data quality has the biggest impact (if you're searching the wrong data source, optimizing the prompt won't help), followed by prompt optimization, then model selection, with hyperparameter tuning last. One concept worth mastering is eval-driven development: write the eval before you build the feature, and keep going until it passes. The people who understand product requirements best—product managers, customer support, sales—can describe in natural language what counts as good, and they should be involved in writing the evals.

Start by reading traces, add those two lines of instrumentation code, write one code eval, and run it to see the results. Fifteen minutes of reading real output will teach you more about your application than an hour of building dashboards. Evals are infrastructure, not an afterthought—their value compounds over time.

One-line Summary

Use traces for error analysis and build evals with code and LLM judges, so your Agent's iterations form a data closed loop.

Paragraph Summary

The article points out that Agents frequently crash after launch because sporadic test runs are used in place of systematic evaluation, and multi-step tool calls let error probability accumulate along the chain—so you must evaluate intermediate steps and final outputs rather than the execution path. The author advocates reading traces first to do error analysis, using open coding to record phenomena and axial coding to categorize root causes, then prioritizing by severity times frequency. The evaluation system consists of three complementary methods: code evals for deterministic checks, LLM judges for semantic issues (calibrated via meta-evaluation against a human-labeled golden dataset), and human evaluation as the gold standard of quality. The article also emphasizes that custom rubrics should be specific and observable, keep binary options, require the judge to reason before scoring, and beware of anti-patterns such as sensitivity to wording, all-in-one judges, and confusing guardrails with north-star metrics.