Best Practices for Claude Computer Use: A Complete Guide to Resolution, Click Accuracy, and Model Selection

Background: The Evolution of Computer Use Technology

Anthropic first introduced the Computer Use feature in October 2024, marking a significant milestone in the history of large language model development. Unlike traditional RPA (Robotic Process Automation), Computer Use enables AI models to visually perceive screen content just like humans and generate action commands such as mouse clicks and keyboard inputs—without requiring pre-programmed UI interaction logic. The underlying principle involves feeding screenshots as visual input to the model, which then outputs structured action commands (such as coordinate clicks and text input) that an execution layer translates into actual system operations. This "see screen → understand → act" closed loop allows AI Agents to control virtually any graphical interface software without requiring the target application to provide a dedicated API.

Anthropic recently released a comprehensive developer guide that systematically summarizes best practices for using Computer Use and Browser Use features across the Claude model family. This guide covers everything from screenshot resolution configuration to click accuracy optimization, making it essential reading for developers building AI Agent systems.

This article provides an in-depth analysis of the core takeaways from these best practices, helping developers avoid common pitfalls and build more reliable computer control integration solutions.

Resolution and Scaling: The #1 Factor Affecting Click Accuracy

Why Pre-Scaling Screenshots Matters So Much

Click accuracy is the foundation of computer control—if clicks fail to hit their targets, all subsequent operations will fail. Anthropic points out that the single most impactful optimization is surprisingly simple: pre-scale screenshots before sending them to the API.

The reason lies in the API's internal image processing limits. For the Claude 4.6 series (Opus 4.6, Sonnet 4.6, Haiku 4.5), the limit is 1568 pixels on the longest edge with a maximum of 1.15 million total pixels. The latest Opus 4.7 significantly raises these limits: 2576 pixels on the longest edge with up to 3.75 million total pixels.

Images exceeding these limits are silently downscaled, causing the model to predict click positions based on degraded images while the developer's coordinate system remains aligned to the original resolution—this is the primary cause of inaccurate clicks at high resolutions.

Deep Dive: Image Token Consumption Mechanism

In Claude's multimodal processing, images are divided into fixed-size tiles for encoding, with each tile consuming a fixed number of tokens. Higher resolution means more tiles, more token consumption, and linearly increasing inference latency and API costs. At 1280×720 resolution, for example, token consumption is approximately 1/9 of native 4K resolution. This is why Anthropic recommends finding the optimal resolution between accuracy and token budget rather than blindly pursuing maximum resolution—images exceeding API limits not only fail to deliver higher accuracy but actually introduce coordinate offsets due to silent scaling, while also wasting transmission bandwidth.

Recommended Resolution Options

Anthropic provides clear resolution recommendations:

Claude 4.6 series: Start with 1280×720 as the default, using approximately 80% of the pixel budget—safe and practical
Opus 4.7: Start from 1080p for a better balance between token consumption and performance
Advanced approach: Use the compute_max_api_fit function to dynamically calculate optimal resolution based on the source image's native aspect ratio, maximizing pixel budget utilization

Resolutions to specifically avoid include: unscaled native resolution (the most common source of accuracy issues), resolutions below 960×540 (too much detail loss), and the hidden 2x resolution issue on macOS caused by a device pixel ratio of 2.

The Hidden Pitfall of macOS Retina Displays

macOS's HiDPI (Retina) mechanism introduces a scaling layer between logical and physical resolution. For example, a MacBook Pro with a logical resolution of 1440×900 may produce screenshots at 2880×1800 pixels (device pixel ratio DPR=2). If a developer sets display_width_px to the logical resolution of 1440 but actually sends a screenshot that's 2880 pixels wide, it triggers the API's silent scaling, causing systematic coordinate offsets. The solution is to always use actual pixel dimensions (not logical dimensions) as parameters, or uniformly scale screenshots to the target resolution before sending.

Coordinate Scaling Mapping Must Not Be Overlooked

When you scale a screenshot before sending it to the API, the click coordinates returned by the model are based on the display resolution you specified. You must proportionally map these coordinates back to the actual screen resolution:

scale_x = screen_w / display_w
scale_y = screen_h / display_h
screen_x = int(api_returned_x * scale_x)
screen_y = int(api_returned_y * scale_y)

This step seems simple, but omitting it or mismatching parameters will cause systematic offset on every click.

Message Construction: Content Order Affects Click Accuracy

An easily overlooked but genuinely effective optimization is the ordering of message content. Anthropic explicitly recommends: when building the content array in messages, place text instructions before images.

# Recommended: text first, screenshot after
content = [
    {"type": "text", "text": "Click on the Submit button"},
    {"type": "image", "source": {...}},
]

The logic is straightforward: let the model know what to look for before processing the screenshot, thereby improving click accuracy. This design closely mirrors human visual attention mechanisms—when we know we're looking for a "Submit button," our visual system prioritizes scanning for areas matching that description rather than processing the entire screen indiscriminately. In Transformer architecture, the prior input of text instructions influences the attention weight allocation for subsequent image tokens, causing the model to establish task-oriented attention bias while encoding visual information. This is a zero-cost optimization that every developer should adopt immediately.

Model Selection Strategy for Claude Computer Use

Positioning and Use Cases for Different Models

Based on internal testing, Anthropic provides clear model selection guidance:

Model	Characteristics	Recommended Scenarios
Sonnet 4.6	Highest mechanical click accuracy, more robust to heavy scaling	First choice for most tasks, best cost-effectiveness
Opus 4.6	Stronger reasoning ability, slightly lower click accuracy	Scenarios requiring complex decision-making
Opus 4.7	Click accuracy matches Sonnet 4.6, strongest reasoning ability	High-resolution source images + complex reasoning
Haiku 4.5	Lowest latency	Latency-sensitive scenarios

Orchestrator + Sub-Agent Architecture: The Optimal Solution for Complex Workflows

A notable advanced pattern is the Orchestrator + Sub-Agent architecture: use a model with strong reasoning capabilities (like Opus) for planning and decision-making, while Sonnet or Haiku handles the actual click operations.

Design Philosophy of Multi-Agent Architecture

The Orchestrator + Sub-agent architecture is the mainstream paradigm in current multi-agent system design. Its core idea derives from the Separation of Concerns principle in software engineering. The orchestrator handles task decomposition, state tracking, and high-level decision-making—for example, determining "should we fill in the form or click next?" The sub-agents focus exclusively on executing specific atomic operations, such as precisely clicking a coordinate. This division of labor not only achieves optimal balance between reasoning quality and execution efficiency, but also provides better observability: the orchestrator's decision chain is clearly auditable, and sub-agent execution results are easy to verify and retry. In practice, this architecture also naturally supports parallelization—multiple sub-agents can simultaneously handle different UI operation tasks, with the orchestrator coordinating results.

This division-of-labor pattern can bring significant improvements in complex workflows, while also effectively controlling overall API costs by concentrating expensive Opus calls at the decision layer.

Four Strategies for Clicking Small Targets

Click accuracy decreases as target size shrinks. Large and medium UI elements (buttons, input fields, standard menu items) perform reliably within safe resolution ranges, but tiny targets like checkboxes, system tray icons, dropdown arrows, and small toggles are challenging.

Anthropic provides four strategies:

Enable zoom functionality: Claude 4.6 and 4.7 support enable_zoom: True configuration, allowing the model to inspect specific screen regions at higher resolution before clicking. This is similar to how humans instinctively lean closer to the screen before clicking small targets—the model first identifies the target's approximate location, then performs localized zoom analysis on that region to obtain more precise coordinate estimates.
Increase target size: If you control the UI being automated, lowering system DPI, increasing browser zoom level, or adjusting UI scaling settings can significantly improve reliability
Use keyboard instead of clicking: For extremely small elements, keyboard shortcuts or Tab navigation are often more reliable than clicking
Optimize source image resolution: Screenshots from 4K+ displays compressed to 720p will shrink a 16px checkbox to approximately 5px—consider using Opus 4.7's higher resolution limits, or capture partial screen regions

Optimization Methods Proven Ineffective (Pitfall Guide)

Equally valuable is Anthropic's sharing of optimization methods that their testing found to be ineffective:

Sending images in chunks: Splitting screenshots into quadrants or regions and sending them separately did not improve click accuracy
Overlaying coordinate grids: Adding visual coordinate grids on screenshots produced no reliable gains
Scaling algorithm selection: Common scaling algorithms like PIL LANCZOS and sips showed completely identical results

Why These "Intuitively Correct" Methods Don't Work