Do Larger LLMs Handle Errors Better? MCP Fault Injection Experiment Says No

Tool-calling LLM Agents are becoming the dominant form of AI applications, and the Model Context Protocol (MCP) is increasingly used as the "socket" connecting models to external tools. But existing benchmarks almost universally assume tools return results on time and correctly—an assumption that doesn't hold in the real world. A new study systematically examines Agent resilience under real MCP faults through protocol-aware fault injection, and reaches a counterintuitive conclusion: model scale does not buy fault tolerance.

Why Existing Evaluations Mask Agent Fragility

Current mainstream capability benchmarks (such as tool-calling accuracy and task completion rate) are mostly measured under the ideal condition that "tools always work." This effectively only examines how Agents perform in the lab while ignoring common production conditions like tool timeouts, errors, and intermittent failures. The study points out that evaluating tool-calling Agents on capability benchmarks alone systematically masks their operational fragility.

In other words, an Agent that scores high on clean benchmarks may have its entire task chain collapse in real deployment due to a single tool call failure. This gap between "high benchmark scores, production failures" is exactly what the study aims to quantify.

Tool Calling refers to an LLM outputting structured function-call instructions while generating text, which an external runtime executes and returns results to the model for continued reasoning. Unlike traditional text completion, tool calling requires the model to understand tool schemas, correctly fill parameters, and maintain call state across multiple turns. Model Context Protocol (MCP) is an open protocol proposed by Anthropic in late 2024 that standardizes how models connect to external tools and data sources using a client-server architecture, enabling the same tool definitions to be reused across different models and frameworks. MCP servers expose primitives such as tools, resources, and prompts via JSON-RPC, and Agents initiate calls through MCP clients and wait for responses. Precisely because MCP abstracts tool calls as network requests, distributed-system failure modes inherent to such systems—timeouts, connection drops, server errors—are introduced into the Agent's execution path.

MCP-Injector: Bringing Real Faults into the Test Pipeline

The research team built MCP-Injector, a drop-in, protocol-aware fault injection middleware. It intercepts the MCP tool-call path and injects controllable fault patterns, supporting three persistence types: permanent, transient, and intermittent, along with reproducible seeded scheduling.

More compelling is how the fault model was calibrated: researchers mined the real ecosystem's fault distribution (fault mode × persistence) from 57,240 issue and PR discussions across 1,774 open-source MCP repositories, then designed injection conditions accordingly. This means the faults used in experiments weren't fabricated but derived from actual incident profiles in the public MCP ecosystem, with an overall stress profile weighted toward permanent faults.

Fault Injection is a classic reliability engineering technique that deliberately introduces errors into a system to observe its behavior and recovery capability, commonly used in distributed systems and chaos engineering. By persistence, permanent faults mean a service remains unavailable until manually repaired; transient faults are brief glitches that self-recover, such as momentary timeouts; intermittent faults are unpredictable failures that appear randomly over a period. Each poses different challenges to Agents: permanent faults test degradation and fallback paths, transient faults test whether retry strategies are sound, and intermittent faults easily induce non-deterministic behavior. Seeded scheduling uses a fixed random seed to control fault trigger timing and sequence, making experiments reproducible and comparable—a key engineering detail for ensuring the credibility of injection experiment results.

Experimental Design: Two Scales × Three Frameworks × Three Fault Types

The experiment comprised 1,350 test units, crossing three dimensions:

• Backbone model scale: two tiers of gpt-oss models, 20B and 120B;
• Agent frameworks: a custom multi-turn plan-and-execute, OpenAI Agents SDK, and LangChain;
• Fault injection conditions: combinations of three fault conditions.

Evaluation metrics included task completion score (out of 10) and resilience ratio (the proportion of fault-free score retained under faults), among others.

Key Finding 1: Faults Cause a Steep Quality Drop

Under this stress profile weighted toward permanent faults, task completion scores plummeted from 8.08 without faults to 3.23, a drop of about 60%. This magnitude shows that when the tool-call path experiences faults matching real-world distributions, Agent task completion capability is severely impacted—not merely "slightly degraded."

For engineering teams, the implication is direct: if your Agent relies on MCP toolchains for critical tasks, tool-side stability directly determines end-to-end success rate; the model's intrinsic capability ceiling isn't the bottleneck.

Key Finding 2: Scale Doesn't Buy Fault Tolerance

The study's most notable conclusion negates the intuition that "larger models are more reliable." Across the two gpt-oss backbones tested:

• The 120B model did not retain a higher proportion of fault-free score under faults than the 20B model; the resilience ratio model yielded p = 0.96, meaning the scale difference is not statistically significant;
• In terms of completion rate under faults, 120B was actually slightly lower: 91.1% vs 95.6%, a 4.4 percentage point difference (95% confidence interval [-0.2, 9.3]).

In other words, moving from a 20B to a 120B model did not make Agents more robust in the face of tool faults. The researchers therefore propose that fault tolerance comes more from system-level design than from stacking up model parameters.

A p-value indicates the probability of observing the current or a more extreme result if the null hypothesis is true; p=0.96 means there is almost no evidence supporting a difference between the two model tiers. A confidence interval gives the plausible range of the effect size; [-0.2, 9.3] crosses zero, indicating that the completion rate difference between 120B and 20B cannot be statistically distinguished from random fluctuation. Note that statistical non-significance does not prove the two are exactly equal, but rather that no difference was detected at this sample size; however, combined with the direction of the point estimate (the larger model was actually slightly lower), the conclusion that "scale doesn't buy fault tolerance" has practical reference value. The resilience ratio, measured as the ratio of score under faults to fault-free score, removes the interference of absolute capability differences and enables cross-model-scale and cross-framework comparison.

Key Finding 3: Framework Fault Tolerance Differences Matter More Than Scale

The three Agent frameworks differ in their ability to absorb faults, suggesting that framework-level error handling mechanisms are more critical than model scale. Explicit planning structures like multi-turn plan-and-execute, SDK built-in retries and exception catching, and the error propagation patterns of chain-based frameworks all substantively affect final performance under faults.

This finding has direct reference value for architecture selection: rather than blindly pursuing larger backbone models, it's better to first examine whether the Agent framework has sound retry strategies, degradation paths, and fault isolation capabilities.

Methodological Insight: Fault Injection Should Be a Complementary Evaluation Dimension

The study's greater contribution lies in proposing an evaluation approach—protocol-aware fault injection. It expands the evaluation focus from "can the model get it right" to "can the system hold up when things go wrong"; only combining both can reflect performance under real deployment conditions.

For teams building MCP tool ecosystems or Agent products, feasible practices include: building fault profiles from historical fault records of your own toolchain, running fault injection tests regularly in CI pipelines, and making resilience ratio a release metric alongside task accuracy.

Practical Recommendations

Synthesizing the conclusions above, several actionable directions emerge:

1. Don't pin fault tolerance hopes on switching to larger models—no positive correlation was observed between scale and fault resilience;
2. Prioritize hardening framework-level error handling—retries, timeouts, degradation, and idempotent design should be default configurations of the Agent stack;
3. Drive testing with real fault distributions, rather than constructing idealized exception cases;
4. Include fault tolerance metrics in release gates to prevent high capability benchmark scores from masking production fragility.

The reliability of tool-calling Agents is ultimately a systems engineering problem. Model capability determines how far it can go, while fault-handling design determines how steadily it can go in the real world.

One-line Summary

Real MCP fault injection experiments show Agent scores drop by 60%, and model scale doesn't buy fault tolerance.

Paragraph Summary

This study systematically evaluates tool-calling LLM Agent resilience under real MCP faults. The team built MCP-Injector, a protocol-aware fault injection middleware whose fault model was calibrated from 57,240 issue/PR discussions across 1,774 open-source MCP repositories, covering permanent, transient, and intermittent persistence types. The experiment crossed 20B and 120B gpt-oss models, three Agent frameworks, and three fault conditions across 1,350 test units. Results show that a stress profile weighted toward permanent faults dropped task completion scores from 8.08 to 3.23, a decline of about 60%; model scale had no significant effect on resilience ratio (p=0.96), and 120B had a slightly lower completion rate under faults (91.1% vs 95.6%). Framework-level error handling differences determine performance under faults more than model scale, leading the study to advocate fault injection as a complementary evaluation dimension alongside capability benchmarks.