Interfaces Screen Readers Can't Read Are Training Your AI Agent

Paper quantifies accessibility labeling gaps inherited by computer-use agents, finding a 39.5 percentage point coverage difference between old and new datasets.
The study is the first to propose and quantify 'accessibility debt': computer-use agents and screen readers share the same accessibility tree, so agent training data inherits apps' missing labels. Across over 17.37 million elements in two datasets, 2017's Rico had only 17.74% coverage vs. 57.27% for 2024's AndroidControl—a 39.5 point gap robust across five coding rules. Input controls are worst. Old data has 5x more placeholder descriptions. Demonstration data shows a 9.0 point selection bias toward labeled elements.
Computer-use agents learn to operate mobile interfaces by analyzing screenshots and accessibility trees. That same accessibility tree is also what screen readers use to interpret interfaces for visually impaired users. This means agents inherit any labeling gaps present in the underlying apps during training—a phenomenon a new paper calls "accessibility debt" and quantifies for the first time.

Two Datasets, a 39.5 Percentage Point Gap
The study measured content-description coverage across two Android interface datasets: Rico, collected in 2017, and AndroidControl, collected in 2024. Across more than 17 million interface elements, using the authors' coding rules, only 17.74% of elements in Rico that required content descriptions actually had them, compared to 57.27% in AndroidControl—a difference of 39.5 percentage points.
This gap held across five alternative coding rules and after resampling entire apps, indicating it's not an artifact of a particular labeling convention. However, the authors explicitly caution that the two datasets also differ in app selection, collection methods, and platform versions, so the figure describes a difference between datasets, not a direct indication that "accessibility labeling improved over time."
Input Controls Are the Weakest Link
In both datasets, input controls had the lowest label coverage among all role types: just 2.20% in Rico and 27.21% in AndroidControl; counting hint text as a name raises AndroidControl to 30.99%. Input fields are precisely the elements agents interact with most when performing tasks like searching or filling out forms—missing labels mean agents must rely on visual cues to guess their purpose.
Rico is a large-scale Android interface dataset released by Google in 2017, containing about 72,000 screens and nearly 1 billion UI elements, long used in interface understanding, generation, and automated testing research. It was collected by parsing view hierarchies on real devices via crowdsourcing, thus reflecting the accessibility labeling status of many mainstream apps at the time. AndroidControl, released in 2024, is a benchmark dataset for computer-use agents, collected from newer Android versions and accompanied by task instructions and action trajectories. The two differ markedly in collection year, app pool, and interface scale—which is why the authors emphasize that the coverage gap cannot be directly interpreted as a temporal trend.
Accessibility tree is a semantic structure tree built by the operating system from interface components, where each node carries attributes like role, name, content description, and state. Screen readers traverse this tree to read interfaces aloud, and computer-use agents likewise use it to locate interactive elements. Content description is an alternative label manually provided by developers when a component has no visible text, telling assistive technology "what this thing is." Role determines an element's semantic type, such as button, input field, or checkbox. When content descriptions are missing or roles are vaguely defined, screen readers can only announce "button, unlabeled," and agents lose reliable semantic signals, falling back to pixel-based visual guessing.
Placeholder Pollution: Old Datasets' Problems Go Beyond "Missing"
Low coverage is only one side of the problem. The study further found that elements with descriptions in the old dataset were five times more likely to have placeholder descriptions than in the new dataset—10.25% versus 1.95%. In other words, many "described" elements in Rico actually carry filler text developers tossed in, offering no real information to screen readers or agents. This finding rules out the alternative explanation that "the gap is just because developers filled in nonsense," and instead suggests the description quality problem in old data is more severe than coverage numbers indicate.
Selection Bias in Demonstration Data
The paper parsed 26,205 task actions to the specific elements they touched, examining a subtler effect: among clickable but textless graphic elements, those operated in demonstration data had a 9.0 percentage point higher description rate than other elements on the same screen. About 3.5 percentage points of this comes from "which element was selected within the same screen"—i.e., better-labeled elements are more likely to be chosen as demonstration targets.
This is a within-benchmark selection effect: demonstration data naturally skews toward well-labeled elements, whereas interfaces agents encounter in the real world may not. Agents trained on such data may have their generalization ability overestimated when deployed on interfaces with missing labels.
Selection effect in statistics refers to samples not being randomly drawn but systematically biased toward certain types of individuals due to some screening mechanism. In demonstration data collection, annotators or automated pipelines preferentially select elements that are easy to operate and semantically clear, causing the element distribution in training corpora to diverge from real interface distributions. This bias lets models perform well on benchmarks yet fail to generalize to real apps with missing labels. The authors cleverly decompose it into two parts via same-screen comparison: cross-screen differences (which interfaces were selected) and within-screen differences (which element on the same interface was selected), the latter being about 3.5 percentage points—pure bias from collection behavior.
What This Means for Computer-Use Agents
The value of this work lies in connecting two previously separate discussions: accessibility engineering and agent training data quality. Screen reader users and computer-use agents consume the same accessibility tree—the former's humanitarian concerns and the latter's performance concerns converge here. When app developers neglect content descriptions, the harm extends beyond visually impaired users to automated systems that will rely on this interface data for training.
The authors released the full pipeline and conducted double-coder reliability checks (kappa values of 0.91 and 0.96), providing a reproducible foundation for future research. For teams building computer-use agents, this study suggests a practical checklist item: when evaluating training corpora, beyond task diversity and interface coverage, also measure the completeness of accessibility labels—it may directly affect agent robustness on real devices.
Limitations and Open Questions
The paper itself remains restrained about causal inference: the 39.5 percentage point gap spans seven years and two different collection methods, so it cannot be simply attributed to the passage of time. More rigorous conclusions require longitudinal comparisons of the same apps under the same collection protocol. Moreover, content description coverage is only one dimension of accessibility debt; gaps in role semantics, focus order, and state announcements have yet to be measured.
One-line Summary
Paper quantifies accessibility labeling gaps inherited by computer-use agents, finding a 39.5 percentage point difference in description coverage between old and new datasets.
Paragraph Summary
The study is the first to propose and quantify "accessibility debt": computer-use agents and screen readers consume the same accessibility tree, so agent training corpora inherit apps' labeling gaps. Across over 17.37 million interface elements in two major datasets, only 17.74% of elements in 2017's Rico that should have content descriptions had them, versus 57.27% in 2024's AndroidControl—a 39.5 percentage point gap robust across five coding rules. Input controls had the lowest label coverage. Old dataset descriptions were five times more likely to be placeholders. Demonstration data also showed a selection effect: operated elements among textless clickable graphics had a 9.0 percentage point higher labeling rate. The authors remain restrained about causality and release the pipeline and double-coder reliability checks.
Related articles

AI Writes Your Code—Do You Still Need to Learn Git? Five Reasons
AI coding agents handle Git operations automatically—is it still necessary to learn Git? Five reasons: team communication, reviewing agent PRs, monitoring code changes, fallback when AI goes down, and the joy of learning—plus how the focus of learning Git is shifting in the agent era.

Anthropic Reportedly Prices IPO at $83 Per Share, Valuing Company at $1.5 Trillion
Anthropic reportedly prices IPO at $83 per share, above the expected range, with a fully diluted valuation of $1.5 trillion. This article analyzes the signal significance of the pricing and its impact on the AI industry's financing environment.

Open-Source Code Hosting Platform Codeberg Suffers Outage, Sparking Debate in Developer Community
Open-source code hosting platform Codeberg.org suffered an outage, sparking Hacker News debate over nonprofit platform stability and open-source infrastructure dependency risks.