Agent Trace All Green, Yet It Did Nothing: Five Silent Failure Patterns

Title

Agent Trace All Green, Yet It Did Nothing: Five Silent Failure Patterns

Meta Description

An Agent trace all green but it did nothing? A deep dive into five silent failure patterns: retry wrappers swallowing failures, step limits masquerading as success, exceptions stringified, parsers force-casting, and silent model downgrades — plus how to catch them in evals with run-shape assertions.

Article Content

A data sync Agent reported that it had reconciled 1,400 records one morning. The trace was green from start to finish: no exceptions, latency within the normal range, every span closed cleanly. But it hadn't actually reconciled anything. One of its tools failed three times in a row, the retry wrapper around it handed off the third failure's result as an ordinary return value, and so the Agent wrote a confident execution summary on top of an empty payload.

According to the original author's experience shared on rss, not a single link in that chain threw an error. And that's exactly the point worth reminding yourself of over and over: the worst Agent bugs don't throw exceptions. The framework catches the failure for you — that's its job — and the act of catching is precisely what buries the failure.

A Caught Error Is Not a Logged Error

The mechanism is boringly mundane. Tracing marks a span as failed when an exception escapes that span's code block. That part is automatic, and it's the behavior everyone relies on. But once you wrap a risky call in a try and handle the error inside the block, nothing escapes, and so nothing gets marked. The span stays in its default state — the state that is neither success nor failure — and almost every trace UI paints the default state green.

So that retry wrapper that swallowed the tool failure wasn't, strictly speaking, lying. It just never recorded that anything went wrong. An error-rate dashboard built on span status will draw a perfectly clean line over an incident. The failure is real; the signal has already vanished before it reaches you.

There's a fundamental distinction in observability at play here: span status and semantic correctness are two different things. In tracing specs like OpenTelemetry, a span's default status is UNSET, and it only becomes ERROR when an exception escapes that span's code block or code explicitly calls setStatus(ERROR). "Green" doesn't mean the framework actively judged "this call succeeded" — it just means "nobody came to tell me it failed." Most trace UIs render UNSET as green to reduce visual noise, which inadvertently paints "unknown" and "success" the same color.

This also explains why catching an exception inside a span is a dangerous move: it suppresses two independent signals at once — the exception itself, and the status marker that should have followed it. To fix this, you need to re-mark actively inside the catch block, e.g. by calling span.recordException(e) and span.setStatus({ code: SpanStatusCode.ERROR }), then decide whether to resume execution.

Five Predictable Silent Failures

When the author started chasing patterns instead of symptoms, the same handful of recovery habits showed up every time. Each one looks reasonable in isolation, and each one deletes the signal you could have used to pinpoint the problem.

Retry Wrapper Returns the Last Attempt

A call that failed three times becomes an ordinary return value, and the attempt count usually lives somewhere the trace can't see. This is the one the original author says he trips over most often.

Step Limit Reports a Half-Finished Job as Complete

A run that exhausted its step budget returns a structure identical to one that genuinely finished. So "it stopped early" and "it succeeded" look indistinguishable downstream.

Exception Stringified and Stuffed into the Model Context

The error becomes a line of text, the model reads and reasons about it as data, the run keeps going, and your monitoring reads it as healthy.

The danger of this approach is rooted in the "untyped" nature of the LLM context window. When an exception object is converted into a chunk of text via str(e) or JSON.stringify and stuffed back into the conversation history, it's indistinguishable to the model from user input or retrieved documents — all are just ordinary token sequences that can be read, reasoned about, and used as a basis for further answering. The model doesn't know this line of text means "the system has broken"; it extracts information from it like any other context and moves on.

This creates a cascading consequence: the error message not only fails to trigger an alert, it becomes new knowledge for the model. Sometimes the model will even fabricate a plausible explanation based on this error text and write it into the final summary. The safer approach is to route exceptions through a structured channel (such as a dedicated error field, an exit code, or a return object type), making them visible to downstream code — rather than mixing them into a natural-language stream the model can freely interpret.

Output Parser Force-Casts Instead of Erroring

A malformed response gets filled with default values, and downstream code can't tell whether a field was produced by the model or conjured out of thin air by the parser.

Fallback Silently Degrades the Model

When the primary model has trouble, a weaker model silently takes over, the quality drop looks like a prompt problem, and you waste a day fixing prompts that were never the issue.

The Fix: Log First, Recover Second

The author isn't arguing that frameworks should stop catching errors — keeping the run alive is literally why they exist. The problem is that recovery and reporting are two separate jobs, and nearly every default practice above merges them into one, then throws away the reporting half.

So the author changed the defaults, not the behavior itself. The wrapper still retries, but before a final failure returns anything, it records the attempt count and marks the span. The step limit hands back a distinct termination reason instead of a success-shaped object. The broad catch around tool calls narrows into a narrow catch that logs the exception first, then recovers. The parser runs in strict mode, loudly erroring on unknown or missing fields instead of quietly filling them in. When a fallback happens, it records which model actually answered. None of this stops the Agent from continuing to run; what it stops is the Agent lying about what it did.

Find It Through Evals, Not Dashboards

The approach that ultimately worked was abandoning the notion that "the dashboard will expose this problem someday." If output is the only thing you check, then a swallowed failure walks right through the moment the model writes a plausible-looking summary on top of a bad result. Checking output is necessary, and nowhere near sufficient.

What actually catches these problems is asserting on the shape of the run itself — and that can only be done where you know in advance what the run should look like. That means evals, not production. Three checks do most of the work: compare actual step count against the expected range, which cleanly separates a run that hit the step limit from one that finished in four steps; check tool calls, where a tool that was called but never returned a usable result — and a tool that should have triggered but never did — have nowhere to hide; and check the trajectory, i.e. the sequence of actions, which lets you catch re-runs, skipped steps, and loops that final-answer checks can't see.

There's a real ceiling here, and the author is candid about it: if the framework has already erased the signal, no downstream check can reconstruct the original error. These assertions give you symptoms — loud and timely enough to send you back into the trace to diagnose the swallowed exception. Every silent failure the author chased down was found the same way — someone finally noticed a strange answer. It was never a test, and never an alert, and several of them had been running for weeks by the time anyone spotted them. The value of run-shape checks is that you no longer have to depend on that someone.

"Run-shape assertions" are the concrete application of eval-driven development in the Agent context. The biggest difference from traditional output evals is that what's being checked is not what the Agent said, but what the Agent did. "Shape" here refers to the macrostructural characteristics of a single run — which tools were called, in what order, how many times each, which range the total step count falls into, and whether the expected branches were triggered.

Such assertions are feasible in an eval environment but often infeasible in production monitoring because the precondition for shape assertions is that you know in advance what a correct run looks like. Production traffic contains wildly varying tasks with no stable reference frame; an eval set, by contrast, is constructed by you, and each item corresponds to an expected execution path. Once the shape is written into an assertion, any deviation — one extra call, one missing call, a step count out of bounds — fails immediately. The cost is the maintenance overhead of the eval set; the payoff is moving the moment of discovery from "someone happened to notice weeks later" forward to the CI stage.

Five Default Behaviors to Change

If you change only five things, change these. Make the retry wrapper log the attempt count and mark the span on the final failure; give the step limit a distinct termination reason so budget exhaustion never wears the shape of success; do narrow catching around tool calls, logging the exception before recovering; make the parser run in strict mode, failing loudly on missing fields instead of fabricating them; and when a fallback silently takes over, record which model actually answered.

One-line Summary

The most dangerous Agent bugs don't throw exceptions: failures get swallowed by recovery logic, and the trace stays all green while the task spins its wheels.

Paragraph Summary

The article opens with a real case: a data sync Agent reported completing reconciliation of 1,400 records, with an all-green trace from start to finish, yet it reconciled nothing — one tool failed three times in a row, the retry wrapper handed off the third failure as an ordinary return value, and the Agent wrote a confident summary on an empty payload. The author notes that tracing only marks failure when an exception escapes a span, while errors handled inside a try block leave the span in its default green state. He identifies five predictable silent failures: retries returning the last attempt, step limits masquerading as completion, exceptions stringified into context, parsers force-casting, and silent model downgrades. The core fix is to separate recovery from reporting, and to catch these issues with run-shape assertions in an eval environment rather than relying on production dashboards.