AI Agents Failing in Production? Build an Evaluation System in Four Steps

Why do so many AI agents dazzle in demos but fall apart in production? Google engineer Dani, joined by open-source developer Matt on the AI Agent Clinic show, answered this in a 60-minute live demo: the lack of a systematic evaluation system. Using a documentation-generation agent (Docs Agent) as the example, they showed how to build an evaluation pipeline from scratch in four steps—and how to expose real quality issues in a LangGraph-based agent before deployment.

The Missing Evaluation: The Hidden Trap of Shipping Agents

Matt is the developer of the open-source documentation tool Docs Agent, an agent built on LangGraph that automatically analyzes GitHub repositories, finds documentation gaps, and generates fix PRs. During the demo, Dani admitted: "If you asked me to evaluate your agent, it would be a bit difficult." When developers judge whether an agent is working purely by feel, and after changing code can only eyeball whether the output "looks right," quality regressions become nearly impossible to detect.

This pain point is universal. As the video points out, nobody tells you what data to test with, and nobody tells you how to actually use evaluation tools. As a result, many teams either skip evaluation and ship anyway, or pile up a mountain of metrics that can't actually be measured. Dani's advice is the opposite: start with a few core metrics, such as tool-use quality and trajectory quality.

Building an Agent Evaluation System in Four Steps

Step 1: Have a Coding Agent Understand the Source Code and Traces

The first step, as Dani emphasizes, is to "point Anti-Gravity at the agent's source code and let it understand how it works internally." In the demo, they directly instructed the coding agent to analyze the trace data from a run, understand how the agent works, and generate more test scenarios.

Anti-Gravity can not only read code but also render Mermaid architecture diagrams, visually presenting the agent's database and call chains. Matt recommended projects like OpenCode, T3code, and PI Agent as additional test scenarios, so the agent can run Docs Agent on these repositories in a repeatable way, thereby accumulating the data needed for evaluation.

LangGraph and agent execution traces: LangGraph is a stateful agent orchestration framework from the LangChain team. It describes workflows as graphs (nodes and edges) and supports checkpoints, interrupt/resume, and persistent memory. Its traces are usually organized around a single "run," recording each node's inputs and outputs, state transitions, and tool calls—which maps directly to the "trajectory quality" repeatedly mentioned in the article. Compared with looking only at final output, trajectory-level data reveals whether the agent went down a wrong path, repeatedly called tools, or looped at some node—exactly the problems that are hard to spot by eye during the demo phase but are most likely to be amplified in production. Understanding this makes clear why the first step is to have a coding agent read the trace data rather than looking directly at the output.

Step 2: Unify Trace Format with OpenInference

Evaluation depends on usable data. Different frameworks (LangGraph, ADK, etc.) produce trace data in different formats. Although OpenTelemetry provides an appearance standard, spans and attributes still aren't unified across frameworks. OpenInference acts as a converter, aggregating traces from various agents into one place.

As Dani and Matt explain in the video, once the agent conforms to the OpenInference standard, the evaluation data stays compatible no matter how the internal logic changes. In the demo, the tool automatically converted the traces and mapped the data into a format accepted by the Vertex AI evaluation service. This step cut deployment time from weeks to about an hour.

The relationship between OpenTelemetry and OpenInference: OpenTelemetry (OTel) is an open-source observability framework incubated by the Cloud Native Computing Foundation (CNCF). It defines unified collection and export standards for three types of telemetry data: traces, metrics, and logs, and is widely supported by major cloud vendors and APM tools. In LLM application scenarios, OTel provides a general Span model, but it does not specify how "an LLM call," "a retrieval," or "a tool call" should be represented, which means traces produced by different agent frameworks cannot be directly compared at the semantic level.

OpenInference is an extension of OTel semantic conventions initiated by vendors such as Arize, defining a standard set of attributes specifically for LLM/agent scenarios (such as llm.model_name, tool.name, retrieval.documents), and providing instrumentors for frameworks including LangChain, LangGraph, LlamaIndex, and CrewAI. Its value lies in making traces from different frameworks follow the same "vocabulary," so they can be consumed by the same evaluation tools. This is the same thing the article emphasizes: "the evaluation data stays compatible no matter how the internal logic changes"—standardization happens at the semantic layer, not just the transport layer.

Step 3: Turn Subjective Quality into Runnable Metrics

This is the most illuminating part of the entire demo. Matt proposed a reproducible pattern: use AI, under human expert guidance, to rapidly create evaluation metrics. Based on Docs Agent's code, the LangGraph code, and internal operating mechanisms, the agent pre-selected a set of managed metrics while also generating custom judging criteria—structured format, actionability, technical accuracy, code-block completeness, and more.

Dani particularly praised the "code block" metric, because many documents fail to generate code blocks when they should. He also offered advice: don't start with 2,000 metrics. Begin with basics like tool-use quality and trajectory quality, then sit down with developers and users to discuss what quality actually means to them.

The applicable boundaries and common pitfalls of LLM-as-judge: LLM-as-judge means using one LLM to score other LLMs' outputs according to a rubric, replacing human review. It is low-cost and highly scalable for semantic metrics (such as actionability, technical accuracy, and tone consistency), which is why it has become a mainstream approach to agent evaluation. But it has several known biases to guard against:

• Position bias: in pairwise comparisons, it tends to prefer the first or second option;
• Length bias: it tends to give longer answers higher scores;
• Self-preference: models from the same source score each other higher;
• Rubric drift: when the rubric is vague, scores vary widely across batches.

Practical mitigations include: pinning the judge model version, providing positive and negative examples in the rubric, using deterministic metrics (such as whether a code block exists or whether the format is valid) as cross-checks, and periodically sampling the judge's own outputs for human review. This also explains why the article keeps both deterministic metrics and LLM scoring—only when the two types of metrics back each other up can "quantified feedback" be truly trustworthy.

Step 4: Run the Evaluation and Quantify Improvement Directions

The final step is to actually run the evaluation and get quantitative feedback. In the demo, the agent generated an EvalSet JSON dataset, an EvalConfig YAML configuration file, and produced a visualization report in Google colors. The report includes both deterministic metrics (such as latency, token usage, and cached tokens) and LLM-as-judge quality scores.

The results were surprising: Docs Agent's documentation quality score was low, and it also exposed a rendering prompt bug. Matt admitted this made him "a bit eager to go back and fix the agent," but more importantly, he now has a data basis—he can determine whether the code needs improvement or the evaluation method itself needs adjustment.

Is Evaluation for Everyone?

Dani raised an open question at the end. Matt's answer was quite pragmatic: if you're trying to ship a product to production or use it in an enterprise, evaluation is very important; but for independent developers, evaluation is only worth the investment when the agent really needs to perform well in production. He also noted that if the model gets stronger tomorrow or the testing framework changes, the evaluation system needs to iterate along with it.

The core value of this demo isn't any particular tool, but a way of thinking: turn the quality standards in a developer's head into runnable, reproducible, shareable real data. As Matt said, whether you're editing content yourself or someone else submits a PR, you can measure improvement or even regression—that's the key step for an agent to move from demo to production.