DeskForge
A controllable desktop supervision data generation framework for computer-operation agents. It fuses screenshots, accessibility trees, and window geometry information to produce densely annotated datasets for training visual-language models on UI element grounding tasks.
Timeline (last 90 days)
DeskForge proposes a controllable desktop environment data generation scheme that generates supervised data at scale by composing and exploring real applications.
DeskForge fuses three information sources—screenshots, accessibility trees, and window geometry information—into dense element-level annotations.
DeskForge systematically varies scenes across multiple dimensions including application state, content, window layout, appearance, and resolution.
All four fine-tuned models achieved improvements on held-out desktop test conditions and all five external GUI grounding benchmarks.
DeskForge's framework code, dataset, and fine-tuned models are all publicly available on the project homepage.
All Facts (5)
DeskForge proposes a controllable desktop environment data generation scheme that generates supervised data at scale by composing and exploring real applications.
50%UnverifiedDeskForge fuses three information sources—screenshots, accessibility trees, and window geometry information—into dense element-level annotations.
50%UnverifiedDeskForge systematically varies scenes across multiple dimensions including application state, content, window layout, appearance, and resolution.
50%UnverifiedAll four fine-tuned models achieved improvements on held-out desktop test conditions and all five external GUI grounding benchmarks.
50%UnverifiedDeskForge's framework code, dataset, and fine-tuned models are all publicly available on the project homepage.
50%