The Blind Spot in Tool-Calling Accuracy Testing: The Missing "No-Tool" Case

Why "Not Calling a Tool" Should Also Be Tested

In the article How to Test Tool-Calling Accuracy in AI Agents, OpenRouter points out that evaluating an AI Agent's tool-calling ability cannot rely solely on whether it called a tool. If a model invokes a function it shouldn't have, it still outputs a tool call, so the fact that "the response contains a tool call" proves nothing in itself. This point seems simple, but it exposes a structural blind spot in most tool-call registries.

In common evaluation frameworks, tool failures are broken down into two major categories: "choosing the wrong tool" and "choosing the right tool but with incorrect parameters"; within parameters, validation is further split into structure and value checks, because "a value that conforms to the schema can still be wrong"—for example, the user wants ord_7281, but the model passes ord_7282, which fully passes schema validation. DeepEval publishes Tool Correctness and Argument Correctness as two independent metrics, and Phoenix also provides a separate tool-selection evaluator, showing that this split has become an industry consensus.

The Two-Drawer Structure: From a Personal Failure Log to Framework Consensus

The article's author, Tally, is a small AI model running on a public ledger and operating a one-person shop. It previously built a "two-drawer" guardrail based on its own failure log: one drawer records cases of choosing the wrong tool, and the other records cases of parameter errors. This structure matches the shape that evaluation frameworks such as DeepEval and Phoenix eventually converged on.

This consistency is itself weak evidence—it shows that what the author named is indeed a real boundary, not a private obsession. For teams building Agent observability, this observation is worth noting: when designing a tool-call log from scratch, wrong tool selection and parameter errors should be recorded in separate tables; otherwise, later attribution analysis will mix them together and cannot distinguish routing problems from parameter-passing problems.

Tool Correctness and Argument Correctness are two independently published evaluation metrics in the DeepEval framework. Tool Correctness measures whether the model selected the correct tool, i.e., tool-selection accuracy; Argument Correctness measures whether, given that the correct tool was selected, the passed parameters match expectations. The significance of separating them is that a failed call may stem from a routing error (wrong tool) or a parameter-passing error (right tool but wrong parameters), and the two require different attribution and remediation strategies. The Phoenix framework also provides an independent tool-selection evaluator, confirming that this split has become an industry consensus.

The Missing Third Drawer: Calls That Should Never Have Been Made

What truly made the author realize the blind spot existed was the "no-tool case" mentioned in the OpenRouter guide. The author's failure registry only recorded "which calls I made and how they failed," but there was nowhere to store "calls I should never have made in the first place." The article gives two specific examples: re-probing a channel that its own on-site notes had already determined to be invalid, and reading the tail of a post for the third time after having already read it twice.

These behaviors are waste, not mistakes. The registry is blind to them because they do not produce rejections or error responses—the call succeeds, it is just unnecessary. In scenarios billed per call or per token, this kind of hidden cost keeps accumulating, yet it is hard for traditional error-rate metrics to capture.

The no-tool case refers to situations where the user's request does not require calling any tool at all, and the model should respond directly in natural language. The opposite is "incorrectly calling a tool." The OpenRouter guide emphasizes that merely checking whether the response contains a tool call cannot verify the quality of the model's decision-making—the model may well make a redundant call when it should have answered directly, and that call may return successfully, thus being counted as "normal" under traditional error-rate metrics. Therefore, when evaluating tool-calling ability, the assessment must include a check of the prior judgment of "whether a tool should be called at all."

A Practical Fix: Add a Line of Necessity Logging per Call

The fix the author proposes is quite lightweight: in the next window period, record one line of log for each call, asking "was this call necessary?" No compose-time checker can fully determine necessity, but one line of log can measure it. Once enough samples accumulate, this number will show whether the waste is a category-level problem or an isolated anecdote.

This idea has direct reference value for teams building Agent systems. Necessity cannot be automatically determined before a call, but after the call it can be retrospectively annotated by humans or a stronger model, forming an independent statistic for the "necessity" dimension. Alongside the existing tool-selection accuracy and parameter accuracy, it constitutes a third observable dimension.

Implications for Agent Evaluation Systems

Evaluation of tool-calling accuracy is moving from "whether a call was made" toward finer layers: tool selection, parameter structure, parameter values, and now the proposed call necessity. The first three layers already have mature framework support, while the fourth remains at the manual logging stage. For teams that rely on Agents to complete real business tasks, this means evaluation metrics need to reserve room for expansion—one cannot assume that "a successful call means it was correct," nor that "no error means no waste."

The author operates itself with a public ledger, recording every income and expense on the page; this extreme transparency makes "waste" quantifiable and open to questioning. For most teams, a public ledger is not necessarily needed, but at least a drawer should be reserved in the logs for "unnecessary calls."

Compose-time checker refers to a component that performs static or rule-based validation of decisions during the Agent orchestration (compose) stage, i.e., before a call is initiated. The author points out that no compose-time checker can fully determine the necessity of a call, because necessity depends on contextual semantics (for example, whether a channel is invalid, whether a post has been read enough), and such information is hard to capture by rules before the call. Therefore, necessity can only be measured as a post-hoc annotation dimension, not as pre-call interception. This also explains why necessity logging belongs to "retrospective statistics" rather than "real-time protection."