Stronger Models, Worse Tools? The Hidden Pitfalls of Tool Calling in Claude's New Models

A Counterintuitive Phenomenon

We tend to assume that newer models are always better than their predecessors. But developer Armin Ronacher, while building the coding tool Pi, ran into a surprising problem: the latest, most powerful Claude models actually performed worse when calling custom editing tools.

According to Armin's report, newer Claude models — not small models like Haiku, but flagship-tier Opus 4.8 and Sonnet 5 — would "hallucinate" extra fields inside the nested edits[] array when invoking Pi's editing tools. The edit content itself was usually correct, but the parameter structure violated the schema, causing Pi to reject the tool call and prompt the model to retry.

A tool call schema (Function Calling / Tool Use) is essentially a JSON Schema specification that describes tool names, parameter names, type constraints, and required fields. In theory, a sufficiently powerful language model should be able to parse any valid schema and generate compliant calls. In practice, however, a model's ability to follow schemas isn't purely a function of "comprehension" — it's heavily influenced by training data distribution. If certain field combinations appear at extremely high frequency in training data, the model develops a tendency to generate hallucinated fields: mapping familiar field structures onto new, mismatched schemas.

This hallucinated-field phenomenon has deep roots in neural network architecture. At the Transformer level, self-attention layers form strong weight associations for high-frequency co-occurrence patterns during pretraining. When a model encounters specific field combinations repeatedly across billions of tokens of code, those combinations form strong activation pathways in the model's key-value memory. During inference, even if the input schema differs from tools seen during training, similar contextual semantics can activate those historical pathways — causing the model to "complete" fields that don't exist in the current schema. This is fundamentally the same mechanism as "hallucinated API calls" in code completion.

In cognitive science terms, this resembles schema transfer: the brain (or model) tends to interpret new situations through existing mental frameworks, even when the fit is imperfect. Crucially, this transfer bias doesn't automatically disappear as model scale increases — larger models have stronger pattern memory, which can actually amplify this bias in certain scenarios. Once common editing-tool combinations like old_string/new_string or path/content are encoded as strong associative features, encountering a new schema with similar structure but different field names causes the model's generation probability distribution to be pulled toward those high-frequency historical patterns. This is the underlying mechanism behind the "invented extra fields" behavior Armin observed.

Occasionally malformed tool calls from smaller models aren't surprising. What genuinely caught Armin off guard was that the problem was worse on newer Anthropic models. Both Opus 4.8 and Sonnet 5 exhibited this behavior, while older models didn't. In other words, the most state-of-the-art models in the family performed worse on this specific tool schema than their predecessors did.

The Root Cause: Reinforcement Training on Specific Tools

Armin proposed a compelling hypothesis: newer Anthropic models were likely trained via reinforcement learning (RL) to better use the editing tools built into Claude Code.

The application of RL in large language model training has evolved from its original use for "aligning human preferences" (RLHF — Reinforcement Learning from Human Feedback, systematized by OpenAI around 2017) to fine-grained optimization for specific tasks. Modern variants — such as RLAIF (RL from AI Feedback) and Tool-use RL — are now widely used by major vendors to improve model accuracy on targeted tasks.

RL training for tool calling typically uses an execution feedback mechanism: the model generates tool call parameters, the system actually executes the call, and the success/failure signal serves as the reward. Unlike RLHF's reliance on human raters, this can be automated at scale. The design of the reward function is critical — if the reward is based solely on "did the call execute successfully" rather than "did the call conform to a general schema specification," the model learns shortcut strategies tuned to specific tools rather than genuinely understanding schema semantics. In RL literature, this is called reward hacking: the agent finds a shortcut to maximize the reward signal that doesn't reflect the behavior the designer actually intended.

In a tool-calling training setup, vendors typically build a "tool use environment" where the model repeatedly attempts tool calls, observes execution results, and receives rewards based on success rates. This approach can dramatically improve accuracy on specific tools, but its side effect — degraded generalization to tools outside the training distribution — is academically known as out-of-distribution generalization failure. It's worth noting that this problem isn't unique to Anthropic: any vendor using a similar training paradigm faces the same tradeoffs, just to varying degrees.

This creates a subtle side effect: when a model is over-optimized for a specific tool interface, its generalization to other non-standard tool interfaces actually degrades. Third-party coding frameworks like Pi use their own custom editing tools, making them more susceptible to this pitfall — the model unconsciously maps familiar field structures onto Pi's schema, generating calls that don't conform to it.

This reveals a deeper tension in the current AI tool ecosystem: model vendors optimize their models for specific tool protocols to maximize performance in their own agent products (like Claude Code), but this "specialization" comes at the cost of generality. For third-party developers, overall improvements in model capability don't necessarily translate into a better experience in their own products.

Different Vendors, Different Tool Design Philosophies

You might not have noticed that different vendors take quite different approaches to editing tool design:

• Anthropic Claude's editing tool uses a "search and replace" mechanism — the str_replace approach;
• OpenAI's Codex uses an apply_patch mechanism, modifying code by applying patches.

str_replace and apply_patch represent two fundamentally different code editing paradigms. str_replace requires the model to provide the exact original text snippet to be replaced and the new replacement content. Its strength is clarity of intent and precise context location, but it demands high string-matching accuracy from the model — if the original file has whitespace differences or minor formatting variations, the match fails.

apply_patch draws from the mature diff/patch toolchain in software engineering (originating from Unix's diff and patch commands, with decades of history). It uses line offsets and context lines to locate changes, resembling git diff format — the model generates a structured patch in unified diff format. Unified diff marks line ranges with @@ and uses +/- to indicate additions and deletions, offering stronger fault tolerance but greater structural complexity. From practical engineering experience, str_replace performs better for small, precise edits, while apply_patch has an advantage for large multi-line refactors — both approaches are also closely tied to how efficiently the model utilizes its context window.

Each mechanism has its applicable scenarios, but once a model has been heavily reinforcement-trained on one approach, its "default generation mode" becomes deeply encoded in its weights and is difficult to correct through simple prompting.

OpenAI has previously stated publicly that their models were specifically trained to use the apply_patch tool efficiently. This confirms that "specialized training on built-in tools" isn't an isolated case — it's standard practice across major vendors.

This also means that for the same editing logic, the optimal tool interface design may be completely different depending on which vendor's model you use — each model's "muscle memory" has already been trained into a specific shape.

The Dilemma Facing Third-Party Developers

This finding raises a sharp question for independent coding tool developers: Should third-party coding frameworks implement multiple sets of editing tools so they can switch to the best-performing one based on the user's chosen underlying model?

From an engineering perspective, this is a pragmatic but frustrating direction, implying:

1. Higher maintenance costs: Developers would need to maintain separate tool implementations for Claude, GPT, and future models, while continuously tracking changes in each vendor's training preferences.
2. Broken abstraction layers: Ideally, tool calls should be model-agnostic — you define a schema, and any sufficiently capable model uses it correctly. In reality, a model's "tool preferences" are deeply coupled into its weights. This runs counter to the Dependency Inversion Principle in software engineering: higher-level modules (tool frameworks) shouldn't depend on the implementation details of lower-level modules (specific models) — but that's exactly what's happening now.
3. Increased lock-in risk: As models are trained to increasingly favor their own vendor's tool protocols, the entire ecosystem may gradually converge toward a handful of de facto standard tool interfaces, further marginalizing smaller developers.

Worth noting here: Anthropic launched the Model Context Protocol (MCP) in 2024, an open protocol designed to provide a unified standard for interactions between AI models and external tools. MCP was open-sourced in November 2024, uses JSON-RPC 2.0 as its communication foundation, and defines three core primitives: Tools, Resources, and Prompts. Its design draws on the success of the Language Server Protocol (LSP) — which dramatically boosted the editor ecosystem by standardizing IDE-to-language-tool interactions. MCP's overall design philosophy is analogous to the USB protocol: any device following the protocol can interoperate. Several development tools including Zed, Replit, and Codeium have indicated support.

However, LSP solved a purely protocol interoperability problem, while MCP faces an additional challenge: the protocol's recipient (the LLM) is a probabilistic system whose behavior cannot be fully constrained by protocol specifications. Even if all vendors adopt the same schema format, different models' "interpretation" of the same schema is still determined by their training history — each model's inherent field-combination preferences will still diverge based on their respective reinforcement training. In this sense, MCP is more like solving the "northbound interface" standardization problem (how tools describe themselves to models) without yet addressing the "east-west interoperability" problem (consistency in how different models interpret the same description). MCP answers "how to describe a tool" but cannot touch the deeper issue of "training preferences already encoded in model weights." This makes the path to standardization far more complex than it appears: protocol-layer unification is a necessary condition, but not a sufficient one.

A Deeper Lesson: The Mismatch Between Capability Gains and Scenario Fit

The value of this case lies in how it punctures a common myth: a model's "higher benchmark score" does not mean "better performance in all scenarios."

As model vendors use reinforcement learning to "sculpt" their models to fit their own products ever more precisely, those models actually become "narrower" in certain dimensions. Their degraded performance on standard third-party tools is fundamentally a mismatch between training objectives and real-world usage scenarios. In machine learning, this is called the capability-generalization tradeoff: deep optimization for specific tasks often comes at the cost of robustness in out-of-distribution scenarios.

Understanding this tradeoff from a quantitative perspective is illuminating. Under the PAC (Probably Approximately Correct) learning framework, a model's generalization error on a target distribution is jointly determined by training error and a complexity penalty term. When reinforcement learning dedicates a large portion of parameter capacity to a specific tool protocol, training error on that distribution drops rapidly — but the generalization bound for out-of-distribution scenarios widens accordingly. A more intuitive lens comes from the Lottery Ticket Hypothesis: sparse sub-networks within deep networks handle different functions, and specialized RL training activates and strengthens sub-networks tied to specific tools while suppressing competing sub-networks needed for general tool understanding — causing structural degradation in overall generalization ability.

From an information-theoretic perspective, a model's "parameter capacity" is finite. When large amounts of that capacity are used to encode fine-grained patterns for specific tool protocols, the capacity available to support general tool-call generalization naturally shrinks. This mirrors the problem of "over-specialization" in human expert training — a surgeon trained exclusively in one type of procedure may be more helpless than a general practitioner when facing a non-standard case. Notably, this problem doesn't automatically disappear as model parameter count grows: while scaling can simultaneously improve both specialized and general capabilities, if the training signal itself has distributional skew, increased scale may amplify rather than reduce this bias — because larger models have stronger pattern memorization and entrenchment capabilities.

For developers building products on top of APIs, this suggests several practical recommendations:

• Always run regression tests before upgrading: Validate new model behavior against your own real tool chains rather than blindly trusting version numbers.
• Align tool schemas with mainstream conventions: Where possible, design your interfaces to reference vendors' publicly documented built-in tool protocols, reducing the space for model errors.
• Follow standardization protocol developments: In the long run, a more neutral and universal tool-calling standard is the fundamental path to avoiding ecosystem fragmentation.

Conclusion

The slightly paradoxical title "Better Models, Worse Tools" precisely captures a real predicament in today's AI agent ecosystem. Model capability improvements are real — but that progress is advancing along each vendor's own product roadmap, not along an open, universal trajectory.

For third-party tools like Pi, and for all developers building on top of large models, there may be no avoiding this reality: you're no longer writing tools for "a model" — you're writing tools for "a model's specific training preferences." How to maintain a product's generality and maintainability amid this fragmentation will be a long-term challenge that every AI application developer must face.

Key Takeaways