Silent Failures in Agentic Tool Interactions: A ToolUniverse Audit Study

When a Tool Call "Looks Successful" — the Truth May Already Be Lost

As agentic AI systems increasingly rely on automated pipelines to chain multiple tools together, a long-overlooked problem has come to the surface: what happens when a tool call appears to return "success," but the information passed back through the API or wrapper layer is actually incomplete or missing — with no notification to the user or the agent?

A study from arXiv (paper ID: arXiv:2609.26836v1) names this phenomenon silent failures. They are called "silent" because neither the user nor the agent realizes a failure has occurred. The system continues passing data downstream and produces seemingly reasonable scientific conclusions — conclusions that may be built on incomplete or even incorrect information.

This risk is especially dangerous in scientific agentic workflows such as biology. Existing research and benchmarks largely focus on macro-level metrics like "task completion" and "task success rate," rarely drilling into the details of agent-tool interactions. This paper aims to fill that gap.

A Systematic Audit of 15 Scientific Tools

The research team built an audit mechanism to identify silent failures in agent-tool interactions. They selected 15 scientific tools integrated in the ToolUniverse environment as their audit targets, while also examining the API documentation and tool documentation associated with those tools.

To be clear, ToolUniverse here serves only as the experimental environment, not as the subject of critique. In other words, the study focuses on the general problem of agent-tool interaction; ToolUniverse simply provides a real, observable scenario for investigation.

The audit process used a combination of "LLM-assisted candidate discovery + automated testing + manual verification": large language models were first used to identify potentially problematic candidates, automated tests were then triggered, and finally each case was manually confirmed. This semi-automated approach ensures broad coverage while avoiding the false positives that come with purely automated detection.

ToolUniverse is a tool integration platform designed for agentic research, aggregating multiple external API tools from scientific domains such as biology, chemistry, and medicine for use by LLM-driven agents in automated pipelines. Typical use cases include genomic data queries, protein structure retrieval, and literature search. A common characteristic of such platforms is that the tools themselves are provided by third-party APIs, which the platform wraps and exposes uniformly to the agent — meaning there is a natural information translation and processing layer between the API layer and the wrapper layer, making it a hotspot for silent failures. Understanding this architecture helps explain why the vast majority of the 91 confirmed failures are concentrated at the API and wrapper layers, rather than at the agent decision layer or final output layer.

91 Failures — Most Hiding in the API and Wrapper Layers

After manual verification, the researchers confirmed 91 failures. These were categorized into 7 failure loci, describing exactly where in the call chain each failure occurred.

The two most frequently occurring failure types were:

• Missing data or fields: Information that should have been present in the tool's return was silently discarded;
• Inconsistent search, filtering, or ranking criteria: Retrieval logic that diverged from expectations, leading to skewed results.

By layer, the distribution was highly concentrated:

• API layer: 51 cases, the majority;
• Wrapper layer: 25 cases.

This distribution reveals a key pattern: silent failures originate upstream in the call chain and propagate downstream. An inconspicuous missing field at the top, passed through multiple tools layer by layer, can ultimately produce a "fully valid-looking" scientific output. The problem is that downstream components have no way of detecting that the foundation of that output has already been compromised — the study calls this the downstream amplification effect of silent failures.

The wrapper layer refers to the adapter code layer sitting between the raw API and the agent's calling interface. Its responsibilities typically include parameter format conversion, return value parsing and normalization, error handling, and secondary filtering of API responses. The wrapper layer exists to reduce the complexity of agents directly interfacing with heterogeneous APIs — but it also introduces additional risk of information loss. For example, a developer might miss certain nested fields when parsing a returned JSON, or erroneously discard valid data at edge cases in filtering logic. Since the wrapper layer's behavior is completely transparent to the agent above it, such processing errors are nearly impossible to detect from the output side. Together with field omissions in the API itself, they form the two primary sources of silent failures.

"Contextual Reliability": A New Framework for Addressing Silent Failures

In the face of these hard-to-detect failures, traditional "task success rate" metrics are clearly insufficient. The paper introduces a new concept — contextual reliability.

The core idea is: the reliability of a tool call should not be judged solely by whether it "returned a result," but by whether it truly delivered complete, consistent, and trustworthy information in a given context. An API call that returns HTTP status code 200 but loses a critical field still counts as a failure under the contextual reliability framework.

Built around this concept, the paper offers several recommendations for mitigation mechanisms across the entire agent-tool interaction pipeline:

• Testing: Proactively construct scenarios to verify whether tools silently discard information under edge conditions;
• Disclosing: Explicitly notify the agent and user when information is incomplete;
• Monitoring: Continuously observe the health of the interaction chain at runtime;
• Measuring: Establish quantifiable metrics to gauge the extent of silent failures.

"Contextual reliability" differs fundamentally from the traditional software engineering concept of reliability. Traditional reliability is usually judged by whether a system crashes or returns a response, with metrics like uptime and error rate. In agentic workflows, however, an HTTP 200 response can carry semantically incomplete data — the system "hasn't crashed," but the information it conveys has already been distorted. Contextual reliability expands the evaluation dimension from "is the system running" to "did the system deliver complete and consistent information in a specific task context," essentially incorporating semantic correctness into the definition of reliability. This shift is especially important for scientific agents, because in research contexts, incomplete data and incorrect data are nearly equivalent in their potential for harm.

Implications for Agentic System Developers

The value of this research lies in bringing a hazard that has long lurked in engineering practice into plain sight. For teams building multi-tool agentic pipelines, several practical conclusions are worth taking seriously:

"Call succeeded" does not mean "information is complete." Conventional signals like status codes and the absence of thrown exceptions cannot guarantee data integrity. Especially in high-accuracy domains like scientific computing and biomedicine, a single silent field omission can directly affect research conclusions.

Upstream problems get amplified downstream. The closer to the source of the call chain a failure occurs, the greater its potential impact. This means auditing and protective measures should be pushed upstream to the API and wrapper layers, rather than only checking at the final output.

Observability needs to cover the interaction layer. Existing monitoring largely focuses on system performance and task completion. The semantic completeness of agent-tool interactions is becoming a new dimension that needs to be specifically measured and disclosed.

As agentic applications move from demos into production environments, these "invisible failures" are likely to become the next battleground in reliability engineering. The contextual reliability framework may well be one of the new standards the industry needs to establish.