Tool Hallucination: The Structural Blind Spot in LLM Agents and Closed-World Defense

Tool-augmented large language model (LLM) agents are rapidly becoming the dominant paradigm for AI applications in production — from function calling to MCP (Model Context Protocol), agents complete complex tasks by invoking external tools. But a newly published arXiv paper, Closed-World Resolution Against Tool Hallucination in LLM Agents, exposes a serious problem that existing defense systems have collectively overlooked: Tool Hallucination.

Agents invoke tools that simply don't exist, or pass parameters that no schema has ever declared. This isn't a failure of tool selection, nor a gap in tool safety — it's a structural blind spot in the entire defense paradigm.

Why All Existing Defenses Fail

Current safety measures for tool calls fall into two broad categories: tool selection, which helps agents pick the right tool, and gating, which constrains what agents can do with real tools. The paper cuts straight to the point: both approaches rest on a shared implicit assumption — the call issued by the agent points to a tool that actually exists.

Hallucinated calls break exactly that assumption. The authors summarize the blind spot in a single sentence: a hallucinated call is, by construction, not a decision made by any gate, and therefore no gate can reject it. In other words, if a tool doesn't exist at all, every mechanism designed to protect real tools has nothing to act on.

From this, the paper draws a key architectural conclusion: hallucination defense must precede any causal gate. This fills a missing link in agent security architecture — you must first confirm that what's being called is real before you can constrain what it's allowed to do.

Causal gating refers to an independent decision component that evaluates, at execution time, whether a tool call is compliant and should be permitted. Typical implementations include permission-policy filters, parameter compliance validators, and rate limiters. These mechanisms share a common assumption: the call being audited targets a real, registered tool — only then does the gate have a logical anchor point for checking behavior. Hallucinated calls bypass this premise entirely: when a model generates a nonexistent tool name or parameter out of thin air, the gating component finds no corresponding policy entry, cannot trigger any rejection logic, and the call either fails silently or propagates to downstream systems with unpredictable side effects. This is precisely the architectural motivation behind the paper's emphasis that "defense must precede causal gating."

Five Categories of Tool Hallucination and the Resolution Rung

To systematically measure this problem, the authors propose a five-category taxonomy of tool hallucination (H1–H5), spanning everything from fabricated tools to illegal parameters. As a reference baseline, they design a mechanism called the Resolution Rung: a training-free, closed-world resolver whose core logic is deliberately simple — registry membership check plus signature check.

The paper emphasizes that the value of this resolver lies not in what it computes, but in where it must sit architecturally — ahead of any causal gate. The authors also characterize an irreducible residue that cannot be eliminated: so-called "borrowed parameters" — calls that are structurally indistinguishable from legitimate ones at the schema level. These edge cases reveal the inherent limits of purely structural checking.

The Closed-World Assumption (CWA) is a classical concept in logic and database theory: any entity not explicitly declared as existing in the system registry is treated as nonexistent. Its counterpart, the Open-World Assumption, allows the existence of unknown entities to remain undecided. LLMs are inherently open-world in their generative behavior — models generalize from training data and can freely construct tool names and parameter combinations they've never seen. The closed-world resolver proposed in this paper forcibly collapses open-world model outputs to the closed-world boundary of the registry: any tool name not in the registry or any parameter not in the schema is classified as a hallucination and rejected. The design is simple but architecturally indispensable; its trade-off is the inability to handle "borrowed parameters" — edge cases that are structurally indistinguishable from legitimate calls.

Empirical Results: Model Scale Doesn't Help

At the heart of the paper is a measurement and benchmarking study. The researchers tested ten hosted models across two calling interfaces, collecting 322 real hallucination instances.

Several data points stand out:

• Fabricated tool calls are heavily concentrated in unconstrained raw JSON interfaces (34 occurrences vs. 3 in structured interfaces). This shows that the degree of interface constraint directly affects hallucination rates.
• Model scale provides no benefit — a 675B-parameter model performs on par with 7–8B models. This overturns the intuition that "bigger is more reliable" and demonstrates that tool hallucination is a structural problem that cannot be solved by adding parameters.

The engineering implication is clear: rather than relying on a stronger base model to self-correct hallucinations, it's more effective to introduce an explicit closed-world resolution layer at the system architecture level.

MCP: A New Hallucination Surface

The research extends further into the Model Context Protocol (MCP). MCP allows multiple servers to be merged into a single namespace, and this merging itself creates a hallucination surface that no single registry can represent.

The authors introduce a second taxonomy (M1–M5) for this context. On real MCP interfaces, they recorded 154 hallucination cases, including cases from frontier models that appeared "clean" on single-registry interfaces. The reason: namespace collisions and shadowing are structural byproducts of the merge operation. When tool names from multiple servers conflict or override one another, ambiguity and hallucinations arise at the system level even when the model itself makes no error.

This is an important warning for the rapidly growing MCP ecosystem: as more and more services are aggregated into unified entry points, hallucination risk doesn't scale linearly — it undergoes a qualitative shift due to structural merging.

Model Context Protocol (MCP) is an open standard proposed by Anthropic in 2024, designed to give LLM agents a unified way to access tools and data sources. Its core design allows a single agent to connect to multiple MCP servers simultaneously, merging tools from different servers into a single callable namespace. This aggregation architecture greatly expands an agent's capability coverage but also introduces namespace collision problems: when two servers each register a tool with the same name, the merged registry can only retain one, and the other is "shadowed." The model cannot perceive this shadowing during inference — it may believe it's calling a tool from Server A while actually executing Server B's tool of the same name, or vice versa. This form of hallucination doesn't stem from model capability limitations; it is a structural byproduct of the system merge operation, and cannot be eliminated by improving model quality alone.

HTB Benchmark: Making Defenses Comparable

To advance research in this direction, the authors release a versioned Hallucinated-Tools Benchmark (HTB). Its goal is to allow any resolver approach to be compared across submissions, establishing a reproducible and cumulative evaluation standard.

For developers and researchers building agent systems, this paper offers three key contributions: it identifies tool hallucination as a structural problem independent of selection and safety; it establishes the architectural principle that defense must precede gating; and it provides a quantifiable benchmark and taxonomy. As agents are deployed at scale, the importance of this kind of foundational security research will only grow.