Conceptcomputer-use agent / Computer Use Agent
Computer Use Agent
An AI agent that automates GUI tasks by observing the screen (screenshots or accessibility trees) and simulating mouse and keyboard operations. It must understand screen content and decide the next action.
Timeline (last 90 days)
Oct 14
In the execution order of a computer-use agent, the tree should come first (listing interactive elements with names, roles and positions), text second (performing OCR on the window to match visible text when the tree has no labels), and screenshots last (for canvases, unnamed icons, or page verification).
Unverified50%