Computer Use
Computer Use is a technical capability that enables AI models to operate computer graphical user interfaces in a manner analogous to human users. Its core mechanisms include perceiving the current interface state through screenshots, identifying UI controls and interactive elements, and executing atomic operations such as mouse clicks and keyboard inputs. This capability allows AI to autonomously complete complex tasks across applications without dedicated API interfaces, with applications in automated workflows, software testing, and intelligent agent systems.
Core Facts
GPT-6 Astra is capable of operating modeling software and game engines, possessing agentic capabilities to view the screen, click on the interface, inspect work, and continue making modifications.
Anthropic introduced the Computer Use feature to allow AI to directly operate desktop applications
OpenAI的Operator项目专注于浏览器自动化操作
Timeline (last 90 days)
In the livestream, the agent encountered friction points such as errors from the Chrome connector, Computer Use requiring repeated session restarts to take effect, and the permissions of the Codex and ChatGPT apps being independent of each other and requiring separate configuration.
After installing the Computer Use plugin (currently supported only on Mac), Codex can control other applications on the computer, such as opening a browser to search for and download content.
This approach uses ChatGPT tasks to organize discrete Pi sessions into a continuously running process, and uses MCP tools and Computer Use to fill capability gaps.
Computer Use's reliability is usually lower than structured tool calling and its speed is slower, but its coverage is almost unrestricted.
Every step of Computer Use requires screenshotting, recognition, decision-making, and execution, making the loop long and thus slow; interface element recognition may go wrong, and pop-ups, loading delays, and dynamic layouts all introduce uncertainty.
Computer use capability allows a model to observe the screen state via screenshots or by reading the interface accessibility tree, and then output action instructions such as click coordinates and keyboard input
Mainstream agent products such as OpenAI's Operator and Anthropic's Computer Use have both introduced HITL checkpoints to varying degrees.
Computer Use capability allows AI models to directly control operating system interfaces like a human operator, which is fundamentally different from function calling, which relies on pre-written API interfaces.
Traditional RPA relies on pre-defined interface structures, whereas Computer Use can generalize to arbitrary previously unseen pages, because the model understands visual semantics rather than fixed DOM paths
Grok Bot 的屏幕操作能力属于「计算机使用（Computer Use）」范畴，Anthropic 的 Claude 3.5 Sonnet 于 2024 年率先发布该能力的公开预览
40 more timeline events
All Facts (20)
GPT-6 Astra is capable of operating modeling software and game engines, possessing agentic capabilities to view the screen, click on the interface, inspect work, and continue making modifications.
90%VerifiedAnthropic introduced the Computer Use feature to allow AI to directly operate desktop applications
90%VerifiedOpenAI的Operator项目专注于浏览器自动化操作
90%VerifiedAnthropic的Computer Use功能允许Claude模型直接操控整个计算机桌面环境，包括移动鼠标、点击按钮和输入文字
90%VerifiedComputer Use is a capability that allows AI models to directly control computer interfaces by taking screenshots to identify screen content and simulating mouse clicks and keyboard input.
90%VerifiedAnthropic于2024年率先在Claude中推出Computer Use功能
90%VerifiedPerplexity's "Computer" feature falls under the category of AI Agent, allowing AI models to operate computer interfaces, including clicking buttons, filling out forms, browsing the web, and performing file operations.
80%VerifiedOpenAI的Operator和Google的Project Mariner也在探索AI操控计算机的类似路径
80%VerifiedAnthropic launched the Computer Use feature in its Claude model, allowing the AI model to directly interact with computer desktop environments.
80%VerifiedThe Computer Use technology was first publicly demonstrated by Anthropic in October 2024 at the launch of Claude 3.5 Sonnet.
80%VerifiedIn October 2024, Anthropic was the first to release the Computer Use feature for Claude, allowing AI to control desktop applications through screenshot recognition and simulated mouse and keyboard operations.
80%VerifiedAnthropic's Claude has added a "Computer Use" feature.
80%VerifiedAnthropic was the first to launch the Claude Computer Use feature in 2024, after which OpenAI released its Operator product.
80%VerifiedCurrent AI agents operating web pages primarily rely on two technical approaches: computer vision (feeding page screenshots into multimodal large models to identify UI elements) and DOM parsing (reading the HTML structure tree to infer element functions).
75%VerifiedOpenAI and Anthropic have released models with computer-use capabilities, such as Anthropic Computer Use and OpenAI Operator.
75%VerifiedAnthropic's Claude launched a native desktop client and introduced Computer Use capability, enabling it to directly operate the user's desktop applications.
75%VerifiedThe Computer Use capability was first commercially deployed by Anthropic in Claude 3.5, and was subsequently followed by OpenAI.
75%VerifiedOpenAI的GPT-4o with Tools和Anthropic的Claude Computer Use是AI Agent方向的代表性实现
75%VerifiedThe underlying technology of GUI agents relies on the collaborative operation of visual language models' screen comprehension capabilities and action planning modules: first performing semantic understanding of screen screenshots to identify interface elements, then having the planning module generate specific instruction sequences for execution.
70%VerifiedComputer Use is an AI capability first introduced by Anthropic in 2024 and subsequently followed up by OpenAI and others, allowing models to operate graphical interface applications like humans.
70%