DeskForge: Training Computer-Use Agents with Dense Supervision from Desktop Environments

The "Visual Grounding" Challenge in Computer-Use Agents

Enabling AI agents to operate computers the way humans do hinges on one core challenge: action grounding — in a desktop scene packed with multiple applications, overlapping windows, and visually similar controls, the agent must accurately determine "what to click." Real-world desktops are far more complex than clean test interfaces — popup menus, floating windows, and look-alike button icons all compete for the model's attention.

The real problem lies in training data. Existing datasets rarely pair these complex scenarios with dense annotations, and they lack the ability to systematically vary scenes under controlled conditions. As a result, models often fail to generalize to real, dynamic desktop environments. A newly published paper on arXiv, DeskForge, directly addresses this bottleneck with a controllable data generation framework.

Action grounding in multimodal AI refers to the ability to map natural language instructions or intent to the coordinates of specific visual elements — for example, translating "click the submit button" into precise pixel coordinates on the screen. Unlike tasks such as image captioning or visual QA, grounding requires the model to output not text, but executable UI interaction locations. While leading vision-language models (such as GPT-4o and the Qwen-VL series) possess powerful image understanding capabilities, they still exhibit notable weaknesses in precise localization within dense UI scenes: similar icons, small controls, and dynamic popups all cause localization errors, and a single mislocalization in a multi-step task chain can trigger cascading failures. This is precisely why grounding is considered one of the core bottlenecks for computer-use agents.

DeskForge: A Controllable Desktop Supervision Environment

DeskForge's core idea is to build a controllable desktop environment that generates supervision data at scale by composing and exploring real applications. Rather than simple screenshots, it systematically varies scenes across multiple dimensions:

• Application state: programs in different running and interaction states
• Content: the actual data and text displayed in the interface
• Window layout: the arrangement, overlap, and positioning of windows
• Appearance: themes, color schemes, and other visual styles
• Resolution: rendering across different screen sizes

More critically, DeskForge fuses three information sources into dense, element-level annotations: screenshots, the accessibility tree, and window geometry information. It also records the actual outcome of each executed action. This multi-source fusion gives every UI element reliable position and semantic annotations — without requiring manual labeling one element at a time.

The accessibility tree (AT) is a structured interface description provided by the operating system for assistive technologies (such as screen readers). It records each UI control's type, name, state, and bounding box coordinates in a tree structure. Compared to raw screenshots, the accessibility tree provides semantic information beyond pixels — such as a button's clickable state, a text field's current content, or the hierarchy of menu items. DeskForge aligns and fuses the accessibility tree with screenshots and window geometry, so that each visual element carries both "what it looks like" and "what it semantically is." This fusion avoids the bottleneck of purely visual annotations that require manual pixel-by-pixel labeling, while also sidestepping the limitation of pure accessibility tree approaches that cannot capture visual occlusion and rendering details.

DeskForge-1M Dataset Scale

Using this environment, the researchers built the DeskForge-1M corpus at a substantial scale:

• 1.2 million annotated desktop observations
• Over 159.7 million element instances

This level of dense annotation has been relatively rare in the GUI domain. Its value lies not only in volume, but in the fact that every data point comes from a controlled composition of real applications — combining authenticity with diversity. This is precisely the supervision signal needed to train robust computer-use agents.

Consistent Improvements Across Four Vision-Language Models

To validate the data's effectiveness, the research team sampled 200,000 grounding examples from DeskForge-1M to fine-tune four vision-language models (VLMs). The results show remarkably consistent improvements:

All four models improved on both held-out desktop test conditions and all five external GUI grounding benchmarks. Taking Qwen3.5-4B as an example:

• Accuracy improved by 11.51 percentage points on ScreenSpot-Pro
• Improved by 10.11 percentage points on OSWorld-G

The across-the-board gains across four different models demonstrate that the supervision signal from DeskForge is broadly applicable — not overfitted to any particular architecture.

From Element Localization to Long-Horizon Task Completion

An improvement in element localization accuracy does not necessarily translate into real-world task success. DeskForge's research goes further, validating gains on long-horizon tasks.

With a fixed planner, fine-tuned action models were able to complete more complex tasks:

• On WebArena-Infinity, Qwen3.5-4B improved from solving 31 out of 119 tasks to 50
• On OpenApps, from 3 out of 100 tasks to 15

The latter improvement is particularly striking — from near failure (3/100) to 15/100, roughly a fivefold increase in task completion capability. This suggests that more accurate visual grounding genuinely accumulates into substantial task success rate improvements across multi-step operation chains.

Long-horizon tasks refer to goals that require executing multiple interdependent steps in sequence — for example, "find a product on a webpage, add it to the cart, fill in the shipping information, and submit the order." Unlike single-step localization, any misstep in a long-horizon task can prevent subsequent steps from executing, and errors compound along the task chain. This is why even modest improvements in single-step grounding accuracy can produce significant multiplier effects on end-to-end task success rates — a single-step accuracy improvement from 85% to 90% can lift overall success on a 10-step task from roughly 20% to about 35%. DeskForge's fivefold improvement on OpenApps (3/100 to 15/100) is a clear manifestation of this nonlinear cumulative benefit.

The Significance of Scalable Supervision

DeskForge's core contribution is demonstrating that controllable composition of real desktop environments is a scalable source of supervision. Compared to relying on manual annotation or constrained synthetic interfaces, this approach retains the complexity of real applications while enabling infinite, programmatic scaling and variation.

For the rapidly evolving direction of computer-use agents, data is often a more fundamental bottleneck than model architecture. DeskForge's framework code, dataset, and fine-tuned models are all publicly available on the project homepage, providing reusable infrastructure for future researchers working on GUI grounding and long-horizon computer use.

Project homepage: https://saidgurbuz.github.io/deskforge/

For developers interested in AI agents, automation, and multimodal models, this work is worth following — it points toward a clear path for generating large-scale, high-quality supervised data using controllable, real-world environments.