Stop Guessing: A Practical Guide to Evaluating and Improving Agentic Applications

Why Your Agent Keeps Failing in Places You Can't See

Building agentic applications is a different ballgame from writing traditional software. Traditional software has deterministic logic—input A goes through a fixed path and produces output B, and when something breaks you can dig through the logs and pinpoint the exact line. Agentic applications don't work that way. The model might fabricate information, call the wrong tool, or skip steps it was supposed to execute. According to Laurie Voss, Head of Developer Relations at Arise.ai, in a course produced in collaboration with deeplearning.ai, these failure modes simply don't exist in traditional software development—and the bigger problem is this: if you can't see where things are going wrong, your only option is to guess at a fix and hope it actually works.

This is exactly why evaluation has become a core skill in AI engineering. Voss makes it clear in the course that building an evaluation system isn't a nice-to-have add-on—it's a fundamental competency every AI engineer must master. Without evaluation, your iterations are blind; with it, every change has evidence behind it.

From Traces to Test Cases: Turning Failures into Reusable Assets

Voss's methodology starts from a refreshingly practical place: don't design test sets out of thin air—start from real failures you find in the agent's execution traces. Every time the agent fabricates information, calls the wrong tool, or skips a step, that trace is a valuable sample. Converting these failure cases into reusable test cases means you can re-run them after every code or prompt change to verify whether the problem is actually solved.

The value of this approach lies in the feedback loop. In the traditional way of working, after fixing a bug, developers can often only manually reproduce it a few times to "feel" like the problem is gone. Test cases built from traces turn that subjective judgment into repeatable checks, giving the agent improvement process an objective reference point.

Trace is the foundational data unit of observability for agentic applications. A single trace typically records the complete execution path of an agent from receiving user input to producing a final response, including every step's model calls, tool selections, parameter passing, intermediate results, and metadata such as latency and token consumption. Unlike traditional logs, traces are organized by request and preserve the causal order between steps, so they can reconstruct "what the agent saw at the time, what decisions it made, and why it got to this point." Mainstream observability tools (such as LangSmith, Langfuse, Arize Phoenix, etc.) all provide trace collection, retrieval, and visualization capabilities. It is precisely this structured record that makes it possible to extract test cases from failed traces: you can pinpoint exactly which step's decision deviated from expectations and solidify that input and expected behavior into a regression test.

Code Evaluators and LLM Judges: Two Tools, One Goal

The course emphasizes two types of evaluation methods. One is code evaluators, which use deterministic program logic to check whether the agent's output meets expectations—suitable for structured scenarios with clear rules. The other is LLM judges, which use another model to assess the agent's performance—suitable for open-ended outputs that are hard to exhaustively enumerate with rules.

One point Voss specifically calls out is worth noting: you also need to check whether these judges themselves are reliable. This is an easily overlooked trap—if the LLM judge's judgments are themselves unstable or biased, then every improvement decision you make based on them is built on quicksand. Evaluating the reliability of your judges is an indispensable part of the whole methodology.

LLM-as-a-judge is an evaluation paradigm that uses a stronger model to score or classify outputs. Common forms include pairwise comparison, Likert scale scoring, and binary pass/fail judgments. It's necessary because many agent outputs (such as how useful an answer is, whether the tone is appropriate, or whether it stays faithful to the retrieved context) can't be measured with exact matching or regex rules. But LLM judges have known systematic biases: position bias (preferring options that appear first), length bias (preferring longer answers), self-preference (preferring outputs from the same model family), and prompt sensitivity issues. Therefore, in practice, mitigation measures are usually needed—for example, swapping option order for bidirectional comparison, providing clear scoring rubrics, calibrating with few-shot examples, and regularly measuring the agreement rate between judges and human annotations. This is exactly the concrete practice behind the article's point about "checking whether the judge itself is reliable."

Let Evaluations Drive Decisions, Not Intuition

With reliable test cases and trustworthy judges in place, the next step is the real improvement loop. The process Voss describes is: use evaluation results to decide what to improve first, then after making changes, use the same set of evaluations to confirm that the agent has actually gotten better.

This order matters. Many teams in practice do the opposite—first make a change based on intuition, then look at a few examples and conclude "seems better now." The problem with this approach isn't a lack of effort; it's that it can't distinguish between "actually better" and "happens to look better." Evaluation-driven iteration makes causality clear: what was changed, which metrics were affected, and whether any regressions were introduced—all of it is documented and traceable.

Regression evaluation is an indispensable part of evaluation-driven iteration. After modifying prompts or tool logic to address a specific failure case, that case passing doesn't mean the overall system improved—the change might introduce new failures in other scenarios. Regression evaluation means running the full test suite after every change, not just the one case that was fixed, and comparing pass rates and metrics across dimensions before and after the change. It shares the same roots as regression testing in traditional software engineering, but is harder: agent behavior is non-deterministic (the same input may produce different outputs), so a single run isn't enough to draw conclusions. Typically, each test case needs to be sampled multiple times with pass rates calculated, or parameters like temperature set to reduce variance. Only by establishing this kind of repeatable, comparable baseline can you distinguish between "actually better" and "happens to look better."

What This Means for AI Engineers

Voss's course, launched in collaboration with deeplearning.ai, takes the form of hands-on labs covering the full chain—from trace analysis, test case construction, and evaluator/judge implementation, to making improvement decisions based on evaluations. For developers building agentic applications, the practical significance of this methodology is that it lowers the cost of trial and error: agent failure modes are complex and hidden, and converging through repeated trial and error is extremely inefficient, whereas systematic evaluation lets every iteration carry clear hypotheses and verification methods.

Treating evaluation as a first-class citizen may well be the most critical divide between agentic applications that "can run a demo" and those that "can go to production."

One-line Summary

Agentic application failures are hidden and hard to spot—build an evaluation system from traces to drive iteration.

Paragraph Summary

Based on Laurie Voss's views in a deeplearning.ai course, the article points out the fundamental difference between agentic applications and traditional software: models may fabricate information, call the wrong tools, or skip steps—failure modes that don't exist in traditional software and are hard to observe directly, leaving developers to fix things by guessing. Evaluation therefore becomes a core competency of AI engineering. Methodologically, you should extract real failure cases from agent traces and turn them into repeatable test cases; evaluation methods fall into deterministic code evaluators and LLM judges for open-ended outputs, and the judges themselves need to be validated for reliability. Ultimately, use evaluation results to decide improvement priorities and verify the effects of changes, so that iterations are grounded in evidence and you avoid intuition-driven fixes and illusory improvements.