Accessibility Tree

Browser automation Agents locate webpage elements and simulate human operations through DOM parsing, visual recognition, or Accessibility Tree.

The accessibility tree is a semantic tree structure built by the browser based on the DOM; it strips away visual style information and retains semantic information such as the role, name, state, and properties of each element.

Compared to the raw HTML DOM, the Accessibility tree filters out a large amount of purely visual style information and retains semantically interactive elements, reducing token consumption while improving Agent perception accuracy.

Codex Computer Use系统同时接收屏幕截图和Accessibility Tree（无障碍控件树）数据来理解界面

业界正在探索的Computer Use优化方向包括本地运行轻量级视觉模型、预测性缓存减少截图频率、利用Accessibility Tree跳过截图进行结构化推理

Screen readers (such as NVDA, JAWS, and VoiceOver) rely on the accessibility tree exposed by the browser to understand page structure.

Mainstream browser agent frameworks such as Playwright Agent and Browser Use generally adopt a solution combining the accessibility tree with visual screenshots to handle complex UIs.

The Accessibility Tree is structured UI state information exposed by browsers for assistive technologies, containing control state attributes such as aria-checked and aria-selected.

Text-based reading technically relies on the system-level Accessibility Tree.

When an application's accessibility annotations are incomplete (such as in some older desktop software or customized web frameworks), the structure-reading approach still needs to fall back to visual recognition as a fallback.

The technical path of reading interface structure is typically implemented through the accessibility tree exposed by the operating system or application framework, such as Windows UI Automation, macOS Accessibility API, and the browser DOM tree.

The accessibility tree describes the role, name, state, and hierarchical relationships of UI elements in a tree structure, decoupled from visual pixel coordinates, so positioning relies on the semantic identity of elements rather than visual position, meaning tests do not break due to layout adjustments.

Compared to screenshots, the accessibility tree has higher information density, consumes fewer tokens, and is unaffected by visual rendering, but some dynamic content or custom components may not be fully mapped.

DOM/Accessibility Tree-based solutions are faster and less costly, but are prone to element localization failures on dynamic pages.

Mainstream strategies for reducing browser agent action spaces include pruning the DOM tree, replacing full HTML with the accessibility tree, and numbering elements and having the model output the corresponding numbers.

The Accessibility Tree is more robust to visual style changes than raw pixel screenshots, because semantic attributes can remain unchanged when a button changes color or position

The Accessibility Tree is a structured interface representation provided by the operating system that abstracts each control into a tree node with attributes such as role, name, and state

Mainstream GUI agent benchmarks such as OSWorld and WindowsAgentArena have begun to incorporate the Accessibility Tree for state verification