Debugging AI Agents in Production: From Guesswork to Evals and Trace

Many teams launch AI agents based on a "gut feeling"—they glance at the input and output, decide it looks about right, and ship it. This approach, which the speaker Ankur calls "running naked on intuition," might work in a demo environment, but once you're in production, you have no idea at which step the agent quietly goes off the rails. In this tech talk shared on Bilibili, Ankur systematically demonstrates how to build a repeatable debugging loop using tracing (Trace), evaluations (Evals), and coding agents (Coding Agent).

Why launching on intuition is bound to fail

Traditional software debugging is easy because the execution path is deterministic: a handleSubmit method throws an error if validation fails, proceeds to the next step if validation passes, and the stack trace tells you everything. Agents are different. Your code often contains just a single line, agent.run(query), and from there whether it calls a tool, performs internal analysis, or keeps thinking is entirely up to the model.

This means that without trace data, you can't even articulate "what it did," let alone pinpoint the problem. Ankur conducted a quick poll during the talk: the vast majority of people validate agents by feel, fewer have an evaluation system in place, and those who dare say their evaluation coverage is sufficient to iterate confidently are very few indeed. This isn't a matter of competence—it's determined by the non-deterministic nature of agents. It's hard to know in advance what you should be evaluating.

Trace: the foundation of agent debugging

Trace records every step in the agent's execution: LLM calls, retrieved documents, tools triggered, and the internal reasoning chain. Ankur gives a typical example: a user asks "where is my refund?", but the agent calls the getOrderStatus tool and then replies "the refund has been issued." From the tool's return value, it seems correct, but what the user really wants to know is why the refund hasn't arrived yet. Looking only at the final output, you can't spot this error; looking at the Trace, the incorrect tool call is laid bare.

With Trace, debugging shifts from "guessing" to "investigating." You can pinpoint the exact stage where the failure occurred, extract a batch of similar cases as a dataset, and then make a targeted judgment: does the prompt need changing? Is the tool itself broken? Or should a guardrail be added?

Trace and observability: In the context of agents, Trace refers to a structured record of a request's complete lifecycle, typically organized as a tree or chain where each node is a span that records the step's input, output, latency, model parameters, token usage, and error information. Compared with traditional logs, Trace's core advantage is that it preserves the causal relationships between steps—you can clearly see which tool's return triggered a subsequent LLM hallucination, rather than piecing together the timeline from a pile of logs with similar timestamps. The industry's common implementation standard is OpenTelemetry's GenAI semantic conventions, and platforms like LangSmith, Langfuse, and Arize Phoenix are all compatible with this format. Trace granularity requires a trade-off: recording too little makes it impossible to locate problems, while recording every raw prompt brings storage costs and privacy risks, so production environments typically redact sensitive fields and set sampling rates.

The eight-step manual debugging loop is too slow

The standard agent improvement process goes roughly like this: discover a production issue → find the corresponding Trace → sample to form a dataset → analyze the root cause → design a solution → build an evaluation → run experiments → ship, and then repeat. This process works, but it's extremely time-consuming—days can be spent digging through Traces and manually comparing results.

Ankur's approach is to bring all data sources together in front of a coding agent: Trace data, existing evaluation results, human annotations, APM monitoring logs, and even the codebase itself. This way, tools like Claude Code, Cursor, and Codex can automatically handle most of the investigation, compressing a process that took days into minutes.

Live demo: a "health check report" for a financial agent

The case in the talk is a financial advisory agent. A user asks "should I invest in a certain company?", and it directly concludes "you should invest." This is clearly problematic—it neither asks about the user's risk tolerance and financial situation nor has any compliance guardrails. The audience's diagnoses included: it should ask the user more questions, the prompt needs adjustment, and there's a lack of baseline validation.

After configuring the CLI and skills, Ankur had Claude Code pull the agent's Traces and analyze them. The results were glaring: about 44% of interactions produced poor summaries, many answers contained fabricated content, the tool redundancy rate was high, and there were cases where "a tool silently failed but the agent pretended it succeeded." These problems are nearly impossible to fully discover by manually digging through logs.

More importantly, the coding agent can not only diagnose but also fix. It built evaluators targeting "silent tool failures" and "numerical traceability," and ran experiments on the original dataset that caused the failures to verify the fixes.

Another historical case better illustrates the value of iteration: a query involving Korean stocks failed entirely, with a baseline pass rate of 0%. The first fix attempt normalized country names, which didn't work—because the underlying tool version was also outdated. After updating the tool version, the third iteration raised the pass rate to about 90%. Still not perfect, but going from "having no idea what was broken" to "knowing what was fixed and what's still missing" is a qualitative difference.

Evaluation isn't a one-time fix—it's continuous monitoring

An audience member asked: since you can just change the prompt to improve the summary, why bother with evaluation? Ankur's answer gets to the core: the value of evaluation isn't in a single fix but in preventing performance regressions. A prompt works well today, but two months later, after a model update or a shift in how users ask questions, performance may quietly decline. Without evaluation, you simply won't perceive this degradation.

On cost, the speaker also offered pragmatic advice: evaluation does incur extra LLM call costs, but platforms usually support sampling ratio controls, so you don't have to run evaluations on 100% of Traces. Also, the choice of evaluation model should serve your scenario—some use cheaper models for initial screening, while others insist on the strongest model to ensure evaluation quality. There's no universal answer, and there's no universal "best evaluation model" leaderboard.

Common types of evaluators: When evaluating agent outputs, three types of evaluators are typically used in combination. The first is deterministic rule-based evaluation, such as exact match, regex validation, JSON schema validity, or whether tool call parameters contain required fields—low cost and interpretable. The second is LLM-as-a-Judge, where a model scores answers according to a rubric; it's suitable for judging subjective qualities like "whether the summary is faithful to the retrieved documents," but it's subject to known issues such as position bias and verbosity preference, so the rubric usually needs to be specific enough and include few-shot examples. The third is human annotation, the gold standard for calibrating the credibility of the first two. In practice, a layered strategy is common: first use cheap rule-based evaluation to filter obvious errors, then use LLM judging for edge cases, and periodically sample for human review. Evaluation results should also be validated for correlation with online metrics (such as user like rates and task completion rates) to avoid the evaluation set drifting away from the real distribution.

Who this approach is for

Ankur repeatedly emphasized that coding agents are not magic—they also drift and can get stuck fixing minor issues like linting. The role of Skills is to keep them focused on the real failures exposed by Traces, rather than repeatedly polishing code style. Humans still need to play the final reviewer role—confirming that the direction of changes is correct, rather than letting the agent "optimize" for a round while feeling good about itself but actually making performance worse.

For teams that have already launched agents but are still relying on eyeballing samples, this loop of "Trace → diagnose → build evaluations → run experiments → ship → continuous monitoring" is currently the most practical path to reducing uncertainty in production.

The division of labor between guardrails and coding agents: A guardrail is a check layer independent of the agent's main flow. On the input side, it can intercept jailbreak prompts and sensitive topics; on the output side, it can verify factual consistency, compliant wording, and whether necessary disclaimers are included. Its difference from evaluation lies in when it runs: evaluation mostly runs offline or in sampled scenarios to measure quality trends, while guardrails execute synchronously on the path of every online request, requiring low latency and a controllable false-positive rate. In the financial case in this article, problems like "giving investment advice without asking about risk tolerance" can be fixed via prompts and are also suitable for guardrail fallback—for example, detecting before output whether it contains specific investment advice and lacks a disclaimer, and if so, rewriting it or routing to a human. Guardrails and coding agents are complementary: the agent is responsible for discovering and fixing systemic defects, while the guardrail is responsible for continuously intercepting high-risk outputs before and after fixes go live.