[KongchangAI]
· 2 min read· 1,069 words

three.dev: Expose AI Agent Failures Before They Go Live

three.dev: Expose AI Agent Failures Before They Go Live

three.dev automatically discovers silent AI agent failures from live traffic and lets coding agents fix them directly.

three.dev tackles "silent failures" of AI agents in production: errors that throw no exceptions but produce plausible yet incorrect content like fabricated refund policies or order statuses, often only discovered through user complaints. It automatically analyzes live traffic to identify behavioral deviations and ranks them by severity, surfacing high-risk issues first. Integration is a single prompt, letting coding agents like Claude Code, Cursor, and Codex generate fixes directly, closing the discovery-to-remediation loop.

AI agents are rapidly moving into production environments, but many teams lack visibility into how they perform in front of real users. Failures are often silent: a wrong refund policy, a fabricated order status — the system doesn't crash, but the problem has already occurred, and teams only learn about it from user complaints. three.dev aims to change this: it automatically surfaces these failures from live traffic, ranks them by severity, and lets coding agents directly generate fixes.

three.dev product page

Silent Failures: The Hidden Trap of AI Agent Production Deployment

Traditional software monitoring relies on exception logs and error codes, but AI agents fail in a fundamentally different way. They rarely throw exceptions; instead, they generate plausible but incorrect content — such as giving users a refund policy that doesn't exist, or fabricating order statuses. These issues trigger no alerts, and teams often only realize them after user feedback. three.dev's positioning is precisely to fill this gap: proactively identifying behavioral deviations in agents from real traffic before users discover the problem.

What Is a "Silent Failure"

A silent failure refers to a situation where the system throws no exceptions, produces no error codes, and logs look completely normal during operation, yet the output is incorrect or harmful from the user's perspective. In traditional software, such issues are relatively rare because logic errors are usually caught by unit tests or exception handling; but LLM-based AI agent outputs are open-ended and probabilistic — the same input can produce semantically fluent yet factually incorrect answers, so their failures naturally tend toward "silence."

Common causes of such failures include: the model misjudging its internal knowledge boundaries (e.g., fabricating policies or order statuses), context retrieval errors (RAG retrieving irrelevant documents), incorrect tool call parameters, and prompts that are inconsistent with business rules. With no crash signal, traditional monitoring systems centered on error rates and exception stacks can barely cover these cases, and teams can only rely on user complaints, manual spot checks, or offline evaluations — feedback loops that are long and sparsely sampled. This is why "AI Observability" has become an independent topic in recent years: it focuses not on whether the service is alive, but on whether the output is correct in business semantics.

Automatic Discovery and Severity Ranking

three.dev's core capability is automatically analyzing live traffic to identify AI agent failure cases and rank them by severity. This means teams don't need to manually write test cases or rules — the system learns what constitutes a "failure" from user interactions on its own. For teams that have already put AI into production, this automated discovery mechanism can significantly reduce manual review burden while surfacing high-risk issues first.

Technical Background of Automated Failure Discovery and Severity Ranking

Automatically determining "what is a failure" without predefined rules typically relies on two types of signals: first, implicit feedback, such as user follow-up questions, retries, escalations to human agents, session abandonment, or expressions of negative sentiment; second, LLM-as-a-Judge, where another model scores interaction traces against business criteria. Combining the two allows low-cost screening of suspicious samples from massive real traffic, which are then handed to humans for review.

Severity ranking involves risk classification: errors involving money, refunds, account security, or compliance commitments get the highest priority, while wording issues that only affect experience get lower priority. Ranking typically combines error type, number of affected users, frequency of occurrence, and business rule weights. Compared to manually writing test cases, this approach of learning failure patterns from real traffic better matches the long-tail distribution, but it also brings issues of evaluation criteria drift and false positives — bias in the scoring model itself directly affects discovery quality, so human calibration and feedback loops are still necessary.

One Prompt to Integrate, Coding Agents Fix It Directly

The process of integrating three.dev is simplified to a single prompt: run a prompt in Claude Code, Cursor, or Codex, and the coding agent gets the information it needs to write fix code. This design connects discovery and remediation, reducing context switching from problem identification to code changes. For teams using AI coding tools, this means faster closed-loop handling of agent behavior issues.

Coding Agents and the "Discover–Fix" Loop

A coding agent refers to an AI programming tool that can read codebases, understand context, autonomously edit files, and run tests — Claude Code, Cursor, and Codex all fall into this category. Unlike ordinary code completion, they can execute tasks across multiple files and iterate based on feedback.

Products like three.dev transform observability data into context consumable by coding agents, essentially structuring "problem reports" into a set of executable fix instructions, thereby shortening the path from identification to code commit. The value of this model lies in reducing manual context switching, but it also requires failure samples to include sufficient reproduction information (input, context, expected behavior); otherwise agents tend to generate patches that look correct but don't actually address the root cause. How to verify fixes and prevent regressions still requires supporting evaluation and regression testing mechanisms.

Market Positioning and Reflections

three.dev received 125 upvotes and 16 comments on Product Hunt, ranking 9th, categorized under Analytics, Developer Tools, and Artificial Intelligence. This indicates that developer interest in AI agent observability is rising. As more teams put AI into production, ensuring its behavior meets expectations will become a key challenge. three.dev's approach of automatically discovering failures from live traffic offers a new way of thinking about AI quality assurance. However, its actual effectiveness still needs validation in more real-world scenarios.

One-line Summary

three.dev automatically discovers silent AI agent failures from live traffic and lets coding agents fix them directly.

Paragraph Summary

three.dev addresses the "silent failure" problem of AI agents in production: these failures throw no exceptions or alerts, instead generating plausible but incorrect content such as fabricated refund policies or order statuses, which teams often only learn about from user complaints. three.dev automatically analyzes live traffic to identify behavioral deviations in agents and ranks them by severity, surfacing high-risk issues first and reducing manual review burden. Integration is simplified to a single prompt, allowing coding agents like Claude Code, Cursor, and Codex to directly generate fix code, closing the loop from discovery to remediation. The product received 125 upvotes and 16 comments on Product Hunt, ranking 9th, reflecting growing developer interest in AI observability — though its actual effectiveness still requires validation in more real-world scenarios.

#AI agent observability#silent failures#three.dev#coding agents#AI quality assurance
Share:

Related articles