A 20-Pixel Click Offset: Screenshots vs. the Accessibility Tree

Computer-use agents must answer one question before executing any click: what's on screen right now?

There are two paths to an answer. Send a screenshot to the model and let it hunt for buttons in the pixels, or ask the operating system directly — because every serious UI toolkit publishes an accessibility tree for screen readers, listing each control's role, name, state, and position.

Most demos choose screenshots, because they work everywhere. But in practice, you need both, and the order matters.

Two visual input methods for computer-use agents

What the accessibility tree actually gives you

A screenshot is a grid of pixels; an accessibility tree is a set of explicit facts:

button  "Export"        enabled   @ (812, 64)
textbox "File name"     focused   value "report-q3"
menuitem "CSV"          @ (830, 140)

That means three things for an agent:

It clicks elements, not guessed coordinates. The agent says "press the Export button," and the OS returns its exact location. No 20-pixel misses on a crowded toolbar.

It knows state. Disabled, checked, focused, expanded, the current field value. From a screenshot, the model can only infer that a button is disabled from it being a slightly lighter gray.

It's cheap to send. A screen with 40 controls is only a few dozen short lines as text. On Claude, a screenshot costs roughly width x height / 750 tokens (per Anthropic's vision docs), so a 1092x1092 image is about 1600 tokens per step. Text can also be searched and cited precisely by the model.

The accessibility tree is a structured UI description maintained by the operating system for assistive technology (screen readers, voice control, automation tools). It's exposed through each platform's native API: UI Automation (UIA) on Windows, AXUIElement on macOS, AT-SPI on Linux, and AccessibilityNodeInfo on Android. App developers populate this tree via system control libraries or explicit annotations (such as ARIA attributes on the web or iOS's accessibilityLabel). Its key value: it describes a control's semantics, not pixels — a button's role name, title, enabled state, and logical position stay stable even if the theme changes, corners get rounded, or shadows are added. Agents can therefore "find controls by name" the way a human does, without relying on visual features. This also means the tree reflects how much effort developers put into accessibility: the more they invest, the richer the information available to agents.

The tree can be empty, and it can lie

The quality of the accessibility tree depends entirely on how well the app implements accessibility. Common gaps include:

Canvas apps. Design tool canvases, games, maps, chart controls. The whole window is one big element with nothing inside.
Remote screens. VNC, RDP, or Citrix windows are essentially just video; the tree stops at the window border.
Electron and Chromium, before being asked. Chromium only builds a full web accessibility tree once it detects assistive technology. On macOS, Electron apps don't expose a tree until a client sets AXManualAccessibility; Chrome itself needs --force-renderer-accessibility. Until then, you're looking at a shell with almost no children.
Custom-painted controls with no names. You get 12 consecutive button "" entries.
Stale positions during animation and scrolling. Act quickly after reading the tree, or read it again.

This is why screenshots stay in the toolbox: they handle anything that paints pixels.

The Chromium accessibility tree lazy build mentioned here is an easily overlooked mechanism. For performance reasons, Chromium doesn't build the full web accessibility tree by default; it only activates when it detects assistive technology signaling queries through platform APIs — on macOS via the AXManualAccessibility attribute, and on Chromium's command line via --force-renderer-accessibility. Electron apps inherit this behavior, so automation tools often get an almost-empty shell on first connection. The way around it: proactively enable accessibility mode before launching the browser or Electron app, or have the agent explicitly request the accessibility tree once at startup to trigger lazy construction. Otherwise you get flaky results — "sometimes it reads, sometimes it doesn't" — and you may mistakenly conclude the app itself doesn't support accessibility.

The screenshot trap: coordinates and scaling

If you do click starting from a screenshot, the most common silent failure isn't the model's vision — it's scaling.

On a Retina Mac, screenshots use physical pixels while clicks use logical points (half). On Windows, 125% or 150% display scaling is the same story. Many agents also downsample before sending the screenshot to the model, so the model's coordinates live in a third coordinate space.

The symptom: clicks consistently land up-left or down-right of the target with a fixed offset ratio. The fix: maintain one conversion function from "the image the model sees" to "screen points," and route every click through it. Test with the four corners of a known window.

Look at the accessibility tree yourself

Before deciding which approach your agent needs for a given app, spend two minutes with a platform inspector:

macOS: Accessibility Inspector (Xcode, Open Developer Tool). Hover any element to see its role, title, and frame.
Windows: Accessibility Insights for Windows, or Inspect.exe from the Windows SDK. Both read UI Automation.
Linux: Accerciser, which reads AT-SPI. Most assumptions assume X11; Wayland deliberately limits what other programs can observe and inject.
Android: adb shell uiautomator dump /sdcard/view.xml && adb pull /sdcard/view.xml. You get a view hierarchy with text, resource IDs, and bounds.

If the inspector shows named buttons and fields, a tree-first agent will be fast and accurate here. If it shows one big panel, prepare for screenshots and OCR.

The execution order that works

Tree first. List interactive elements with names, roles, and positions. Let the model choose by ID.
Text second. When the tree has no labels, OCR the window and match visible text.
Screenshots last. For canvases, unnamed icons, or "is this the right page?" checks.
Verify after actions. Re-read the tree or text after every significant step; don't assume the click succeeded.

Treating screenshots as a fallback rather than the default makes most steps cheaper and more precise, while still covering apps that are entirely custom-painted.

Why this is worth paying attention to

Auten is an MCP server that gives Claude Code, Codex, Cursor, or any MCP client the ability to operate a real screen: your computer (Mac, Windows, Linux) and Android phones connected via USB (beta). Its order of operations is exactly the logic above: the agent first gets a numbered list of real buttons and fields from the accessibility tree, while OCR and screenshots fill in when an app has no accessibility information.

Which apps have you automated that had the worst accessibility trees? Share in the comments.

Additional background

OCR's role here is a "text fallback layer," not general visual recognition. It works by taking a screenshot of the current window, performing text detection and recognition to get each text block's content and bounding box, and then returning the box centers to the model as clickable coordinates. Compared to sending a full screenshot to the model, OCR output is cheaper and its coordinates are more explicit — the model only needs to pick a target from text labels like "File (F)" or "Save As." It fills the gap left by unlabeled controls in the accessibility tree: for example, a "Save" icon on a custom toolbar has no name, but nearby text or the small caption under the icon can still be captured by OCR. The limits: pure icon buttons, non-Latin scripts, low contrast, or decorative fonts reduce recognition rates, so OCR usually coexists with screenshots to handle "scenes that text can explain."

One-line Summary

Computer-use agents should be tree-first and screenshot-fallback, replacing pixel guessing with element semantics.

Paragraph Summary

This article explores how computer-use agents should understand screen content before clicking, arguing they should prioritize the accessibility tree over screenshots. The accessibility tree is maintained by the OS for assistive technology, providing each control's role, name, state, and exact position, letting agents click by element directly, avoiding coordinate drift, and consuming far fewer tokens as text. But tree quality depends on the app's accessibility implementation, and it fails on canvases, remote desktops, unactivated Electron/Chromium, and unnamed custom controls. Screenshots can handle anything that paints pixels, but clicking from screenshots often drifts silently due to Retina physical pixels vs. logical points and system display scaling, requiring a unified coordinate conversion function. The article recommends this order: tree first, OCR text second, screenshots last, and verify after each action.