Computer Use
Computer Use is a technical capability that enables AI models to operate computer graphical user interfaces in a manner analogous to human users. Its core mechanisms include perceiving the current interface state through screenshots, identifying UI controls and interactive elements, and executing atomic operations such as mouse clicks and keyboard inputs. This capability allows AI to autonomously complete complex tasks across applications without dedicated API interfaces, with applications in automated workflows, software testing, and intelligent agent systems.
Core Facts
GPT-6 Astra is capable of operating modeling software and game engines, possessing agentic capabilities to view the screen, click on the interface, inspect work, and continue making modifications.
Anthropic introduced the Computer Use feature to allow AI to directly operate desktop applications
OpenAI的Operator项目专注于浏览器自动化操作
Timeline (last 90 days)
Two questions determine which AI tool to use: is the object of operation a code file or a software interface? Does the target system have a callable API? If the object is code and the system has an interface, use a Code Agent; if the object is an interface and the system has no API, use Computer Use.
For systems with APIs, programmatic calls are more stable and faster; for legacy internal systems without APIs, desktop clients, and interface flows that require visual confirmation, only tools like Computer Use can operate the screen.
A common current practice among engineers is to combine Codex with Computer Use: Codex handles the coding portion (writing scripts, modifying code, building automation logic), while Computer Use handles the execution portion (running what has been written in a real desktop environment).
Computer Use's capability is simulating human operations on the desktop, such as opening financial software to enter data, launching a browser and clicking interface buttons for testing, and copying Excel data into internal systems that expose no open interface.
The technical path of Computer Use is typically: the model receives screen screenshots and outputs action instructions such as coordinate clicks, text input and scrolling, executing in a loop until the task is complete.
Having AI operate real computers requires solving safety and controllability issues such as avoiding misoperations and defining permissions.
Computer-use models rely on vision-language understanding to adapt to interface changes, and are therefore more generalizable, but they also face challenges such as high latency, low fault tolerance and irreversible operations.
Codex provides a Computer Use plugin, whose purpose is to let Codex directly operate the user's computer.
There are two must-install plugins for Codex: Computer Use (which can invoke apps on the computer, automatically take screenshots, and read window contents) and a black plugin for making visualization charts (after the revamp it disappeared from the public interface and can only be found via search; in testing it made a dynamic chart successfully on the first try).
Hermes's second-layer tools need to be configured and enabled as needed, including WebSearch/WebExtract (web search and scraping), Vision Analyze (visual understanding), Video Analyze (video understanding), STT/TTS (voice interaction), Browser Exec (browser automation), and Computer Use (desktop control).
40 more timeline events
All Facts (20)
GPT-6 Astra is capable of operating modeling software and game engines, possessing agentic capabilities to view the screen, click on the interface, inspect work, and continue making modifications.
90%VerifiedAnthropic introduced the Computer Use feature to allow AI to directly operate desktop applications
90%VerifiedOpenAI的Operator项目专注于浏览器自动化操作
90%VerifiedAnthropic's Computer Use feature allows the Claude model to directly control the entire computer desktop environment, including moving the mouse, clicking buttons, and typing text.
90%VerifiedComputer Use is a capability that allows AI models to directly control computer interfaces by taking screenshots to identify screen content and simulating mouse clicks and keyboard input.
90%VerifiedAnthropic于2024年率先在Claude中推出Computer Use功能
90%VerifiedOpenAI's GPT-4o with Tools and Anthropic's Claude Computer Use are representative implementations in the AI Agent direction.
85%VerifiedPerplexity's "Computer" feature falls under the category of AI Agent, allowing AI models to operate computer interfaces, including clicking buttons, filling out forms, browsing the web, and performing file operations.
80%VerifiedOpenAI的Operator和Google的Project Mariner也在探索AI操控计算机的类似路径
80%VerifiedAnthropic launched the Computer Use feature in its Claude model, allowing the AI model to directly interact with computer desktop environments.
80%VerifiedThe Computer Use technology was first publicly demonstrated by Anthropic in October 2024 at the launch of Claude 3.5 Sonnet.
80%VerifiedIn October 2024, Anthropic was the first to release the Computer Use feature for Claude, allowing AI to control desktop applications through screenshot recognition and simulated mouse and keyboard operations.
80%VerifiedAnthropic's Claude has added a "Computer Use" feature.
80%VerifiedThe Computer Use capability was first commercially deployed by Anthropic in Claude 3.5, and was subsequently followed by OpenAI.
80%VerifiedAnthropic was the first to launch the Claude Computer Use feature in 2024, after which OpenAI released its Operator product.
80%VerifiedThrough AI preprocessing, the structured payload takes on the decoding and retrieval tasks respectively, reducing the human agent's cognitive load and trading machine time for people's high-value time.
75%VerifiedComputer Use refers to the capability paradigm of enabling large language models to directly operate graphical user interfaces (GUIs) or complete computer tasks via API calls.
75%VerifiedIn 2024, Anthropic released Claude's Computer Use feature, which understands screen content through screenshots and simulates mouse and keyboard operations using coordinate positioning
75%VerifiedCurrent AI agents operating web pages primarily rely on two technical approaches: computer vision (feeding page screenshots into multimodal large models to identify UI elements) and DOM parsing (reading the HTML structure tree to infer element functions).
75%VerifiedAnthropic's Claude launched a native desktop client and introduced Computer Use capability, enabling it to directly operate the user's desktop applications.
75%