When Agent Fixes Cause New Failures: Observation-Driven Regression Testing

Why Agent Changes Always Break the System in "Invisible" Places

At the AI Engineer conference, Sofia Jakovcic, a forward-deployed engineer from Arize, raised a problem that almost everyone doing agent development has encountered: you submit a seemingly harmless fix, and it triggers a cascade of other failures. She attributes the core cause of this phenomenon to a widespread cognitive bias — agents rarely fail at the code level; they fail at runtime.

Traditional software logic is written down, so you can clearly predict what will happen at runtime before deployment. But agent logic exists only at runtime: a change to a single natural-language prompt can alter how the agent collaborates with sub-agents, the order in which it calls tools, and thousands of possible execution trajectories. Code review only sees one extra line in the system prompt file, but the real behavioral differences live entirely in the trace (execution trajectory).

She used a compelling example: a security engineer asked for one sentence to be added to a customer service agent — confirm before executing any action. It sounds completely reasonable. A week later, evaluation of production data showed the "action safety" metric improved by 17 points — the security engineer's judgment was correct. But in the same evaluation suite, "task completion" also dropped by 17 points. The reason: the agent confirms before every action, and users may leave the session midway, so tasks never get completed. Ironically, averaging the two metrics yields a net value of exactly zero.

This is the essence of the problem: improvement in a single dimension does not equal overall improvement of the agent. And without numerical evaluation data, teams are "releasing based on gut feeling."

The Five-Step Regression Testing Loop

Sofia presented a regression process that starts from production traces, with the core idea: the diff tells you what changed, and the trace tells you what actually happened.

Step 1: Build a Dataset from Real Production Data

No synthetic data needed — if you already have real user queries, real inputs, and real tool calls, pull directly from production traces. The dataset needs to cover four categories of content: representative workflows, known failure cases, edge cases, and workflows that already succeed and work normally. This last point is often overlooked: regression testing must not only verify that the fix works, but also ensure that nothing that previously worked has been broken.

Trace (execution trajectory) is the structured record produced when an agent runs, typically stored as a tree of spans that captures the call chain of a complete request: the root span is the user input, and child spans record each LLM call, tool execution, retrieval, sub-agent delegation, along with their inputs/outputs, latency, and token consumption. Unlike traditional logs, traces preserve causal hierarchy and full context, so they can answer "why did the agent take this path" rather than just "what happened." Mainstream implementations follow OpenTelemetry's GenAI semantic conventions and can be exported and aggregated across frameworks. Because of this, traces serve both as raw material for regression datasets and as the input unit for LLM judge scoring — evaluators typically rate a specific span or an entire trace, not an isolated final answer.

Step 2: Turn Subjective Metrics into Scores with Evaluators

Hallucination, accuracy, tool trajectory, step efficiency — these metrics are inherently subjective. Human reviewers can judge whether an output is hallucinated or whether a path is optimal, but to automate this process you need to use an LLM as a judge. Provide the evaluation criteria and have the model rate a span or trace against them on a binary or zero-to-five scale. For deterministic checks, you can pair with code evaluators; for non-deterministic checks, hand them to the LLM judge. Build one evaluator per key metric, and you have a test suite.

LLM as a judge refers to using a (usually stronger) model to make judgments on outputs according to a fixed rubric, replacing human annotation. To make results trustworthy, several engineering constraints are needed: write the rubric as decidable criteria with positive and negative examples; force the output into a structured format (e.g., JSON binary classification or a 1–5 score) for aggregation; swap order or apply debiasing to counter position bias and length bias; and calibrate judge-human agreement (e.g., Cohen's kappa) using a small set of human-labeled samples. In addition, the judge itself is a component that needs version management — changing the judge model or modifying the rubric makes historical scores incomparable, so baseline experiments must be recorded with the judge version. Deterministic checks (such as tool parameter schema or whether a forbidden tool was called) should be handled by code evaluators, which are cheaper and fully reproducible.

Steps 3 to 5: Baseline Experiment, Iterative Comparison, CI Automation

First run the agent over the full production dataset and score it with the test suite — that is the baseline. Then add that one line of change, rerun, and compare results. This process can also be used to verify whether a more expensive model is worth switching to — prove the return on investment with numbers rather than intuition.

Finally, wire the experiment into CI: run the evaluation workflow on every pull request, and block merging if it doesn't pass.

On the Arize platform, this process lets you test a single agent in the prompt playground, or connect to a remote multi-agent system in the agent playground, where every record in the entire dataset is run through evaluation and scored.

Eval gate is the practice of integrating the evaluation suite into continuous integration: run evaluators on a fixed dataset in the PR pipeline, compare each metric against the main-branch baseline, and block merging when a threshold is exceeded. Key design considerations for implementation include: set a tolerance for each metric rather than requiring that it never drops (LLM outputs have inherent variance, so sample multiple times or fix temperature to reduce noise); distinguish hard metrics (safety, compliance, tool-call legality) from soft metrics (tone, conciseness); tier evaluation costs by running a small dataset on every PR and the full dataset before merge or overnight; and record each run's scores, judge version, and dataset version in the experiment log, so that "which commit did this regression start from" is traceable.

Letting Agents Complete the Loop Themselves

The final part of the talk proposed a more forward-looking direction. The current loop is completed by human engineers: observe production traces, evaluate and score, implement changes, validate with experiments, release or roll back. And every failure should become a new test case, so the regression suite accumulates itself and becomes harder and harder to break.

But humans themselves are the bottleneck — still watching dashboards, triaging issues, reviewing PRs. Sofia's second point: what if agents could investigate and propose fixes themselves? She mentioned that Arize provides a public skill repository that lets coding agents connect directly to the observability backend, linking code generated in the IDE with observability data. Otherwise, coding agents "push blind" when they open PRs — they only see static code and don't know what's actually happening in production.

Her closing point: the best agents fix themselves, and the future of observability is no longer just humans understanding agents, but agents understanding themselves.

Coding agent "blind push" refers to agents with the ability to read/write code and submit PRs (such as coding agents running in the IDE or CI) only seeing static content within the repository, with no access to production traces, online metrics, or user feedback, so their modifications are based on code shape rather than real behavior distributions. The way to bridge this is usually through a tool protocol like MCP (Model Context Protocol), exposing the observability backend's query capabilities as tools the agent can call, so it can retrieve failure samples by time window, error type, or a specific trace ID, read evaluation scores, and use them to locate root causes and generate patches. This pattern also introduces new risks: the agent may overfit to the visible failure samples, or modify the evaluation criteria themselves due to excessive permissions, so humans still need to gatekeep the evaluation suite and merge strategy.

One-line Summary

Agents fail at runtime, not at the code level; production traces must anchor an evaluation loop for regression.

Paragraph Summary

The article points out that agent logic exists only at runtime: a single prompt change alters tool-call order and masses of execution trajectories, which static code review cannot predict, so failures often occur in "invisible" places. The author uses a customer service agent example where adding "confirm before acting" raised action safety by 17 points but dropped task completion by 17 points, netting exactly zero — showing that single-dimension improvement does not equal overall improvement. She proposes a five-step regression loop: build a dataset from production traces covering representative workflows, known failures, edge cases, and successful paths; turn subjective metrics into scores with LLM judges plus code evaluators; run a baseline, iteratively compare changes, and wire into CI as a merge gate. Finally, she looks ahead to agents observing and fixing themselves, connecting coding agents to the observability backend to avoid blind PR pushes.