17,000 Runs Tested: How Claude, Codex, and Cursor Choose Their Tools

The Tool Selection Problem in AI Agents

As Claude, OpenAI Codex, and Cursor become increasingly embedded in developers' daily workflows, a core question has emerged: when faced with a specific task, how do these AI agents decide which tool to call? Should they read a file, run a search, execute a shell command, or go straight to editing code?

Behind this seemingly simple question lies the core logic of modern AI agent system design. A recent large-scale empirical study covering 17,000 runs used data to pull back the curtain and answer the critical question: what tools do Claude, Codex, and Cursor actually reach for?

Why Tool Selection Matters

In the agent-driven programming paradigm, large language models don't do all the work directly — they perceive their environment, gather information, and take action by calling a series of external tools. This tool-calling capability is precisely what separates a basic chatbot from an AI coding assistant that can actually get things done.

Tool Calls Define an Agent's Ceiling

The set of tools available to an agent essentially defines its "hands and feet." Common tool types include:

• File reading: understanding codebase context
• File editing/writing: actually modifying code
• grep/search: locating relevant snippets in large codebases
• Shell command execution: running tests, building projects, installing dependencies
• Code execution: validating changes

When faced with the same task, different agents often display strikingly different tool preferences. Some tend to read large numbers of files upfront to build a global understanding, while others take a more aggressive approach — jumping straight to editing and iterating based on feedback. These differences directly affect task completion efficiency, token consumption costs, and the quality of the final output.

The underlying implementation of tool calls typically relies on the "Function Calling" or "Tool Use" protocol. During inference, the model outputs a structured call instruction (containing the tool name and parameters), which is captured by an external runtime, executed, and then fed back to the model as new context — forming a closed "perceive–decide–execute" loop. The key challenge here is that the model must decide when to call a tool, which one to call, and what parameters to pass, all based solely on the tool's text description — without actually executing it. This means the quality of tool descriptions, the granularity of the tool set, and the implicit constraints on tool usage in the system prompt all profoundly shape an agent's tool selection behavior. This is also why different vendors, even when using the same base model, can produce wildly different tool-calling patterns.

The Value of 17,000 Runs

A single test or a small sample is full of randomness. Scaling to 17,000 runs allows researchers to strip away random noise and observe the statistical tendencies of different AI assistants in tool selection. At this scale, empirical data provides significant reference value for understanding agent behavior patterns, optimizing prompt engineering, and improving agent architectures.

Three Major AI Coding Assistants Compared

The three subjects in this study each represent a distinct technical approach in the current AI coding assistant landscape:

Claude (Anthropic)

Coding agents built on the Claude model family — especially when used with Claude Code — are typically known for being cautious and thorough about context. Claude tends to fully understand the codebase structure before making modifications, so it's often more active in calling file-reading tools.

Codex (OpenAI)

OpenAI's Codex and the coding agents derived from it excel at code generation. Its tool selection strategy may lean more toward direct generation and execution, converging on the correct answer through rapid iteration.

Cursor

Cursor is a deeply AI-integrated code editor, and its agent mode features unique engineering optimizations in tool orchestration. It needs to balance search, read, and edit operations within the editor context, and its tool selection is tightly coupled with its productized interaction design.

These three assistants differ fundamentally at the architectural level, which helps explain the root causes of their divergent tool preferences. Claude Code and Codex CLI are essentially "model-driven" agents, where the LLM directly controls tool-calling decisions with relatively minimal engineering intervention. Cursor, on the other hand, is an "editor-native" agent whose tool-calling logic is deeply embedded in the IDE's state management. It can directly sense the currently open files, cursor position, diagnostics, and other editor context, which reduces the need for redundant file exploration. This architectural distinction means that Cursor's tool selection largely reflects product engineering decisions, while Claude's and Codex's choices more directly mirror the model's own reasoning tendencies — meaning the two are actually measuring different levels of capability when evaluated.

Measurement Methodology and Key Findings

The core methodology of this type of research is: run all three agents on a unified task set, record the type, frequency, and sequence of every tool call, and ultimately produce comparable behavioral profiles.

Dimensions Worth Watching

With large-scale statistics, this kind of research can surface insights across several dimensions:

1. Read-to-edit ratio: Some agents spend a large portion of their calls on "exploratory reading," while others move into editing much faster
2. Dependency on search tools: In large codebases, the frequency of grep/search tool usage reflects an agent's strategy for locating problems
3. Aggressiveness of shell execution: Whether an agent tends to run commands frequently to validate results reflects its "trial-and-error" style
4. Patterns in tool call sequences: Each agent forms a characteristic workflow (e.g., a "read → search → edit → verify" loop) with its own distinct shape

Engineering Takeaways from the Data

These differences are not isolated phenomena — they are the result of each vendor's combined choices around system prompt design, tool description standards, and underlying model capabilities. For developers, understanding the tool preferences of the assistant you use helps you:

• Better organize your codebase structure to match the agent's exploration habits
• Anticipate potential token costs ahead of time
• Guide the agent toward more efficient strategies through prompting when needed

Token cost is an easy-to-overlook but critically important economic dimension of tool-calling behavior. The content returned by each tool call (such as a full file or command output) gets appended to the conversation history as context, directly accumulating input token consumption. A "read-heavy" strategy means the agent has burned through a large chunk of its context window before it even starts modifying code — which can lead to context overflow or cost spikes when working with large codebases. By contrast, a "search-first" strategy retrieves only relevant snippets through precise targeting, achieving a better balance between informational completeness and token efficiency. Understanding your agent's tool preferences is, at its core, a matter of managing inference costs — for teams that use AI coding assistants at high frequency, this difference will show up directly on the bill once you're operating at scale.

Real-World Implications for Developers and the Ecosystem

The value of this research goes far beyond the surface-level data of "who called what tool." It reflects a deeper trend: the competition among AI coding assistants is shifting from pure model capability comparisons to an overall battle of agent system engineering.

The intelligence of tool selection directly determines an agent's usability in real, complex software engineering scenarios. No matter how capable the underlying model is, if it makes frequent mistakes in tool orchestration — like repeatedly reading irrelevant files or skipping necessary verification steps — the resulting developer experience will suffer significantly.

For teams building AI agents, this kind of empirical data provides invaluable behavioral benchmarks. It's a reminder that the design, description, and invocation logic of tools matter just as much as the model itself. In the future, observability and evaluation frameworks around tool calling may well become an important niche within the AI coding space.

Conclusion

17,000 runs represent just one slice of the vast behavioral space of AI agents, but they make the otherwise black-box process of tool selection quantifiable and comparable. Whether you're a Claude, Codex, or Cursor user, understanding how these assistants make decisions and take action behind the scenes will help you collaborate with AI more effectively.

As competition among AI coding assistants heats up, transparent research based on large-scale empirical testing will receive increasing attention from the developer community — when it comes to choosing your tools, data is always more reliable than intuition.