Hands-On with OpenAI Codex Computer Use: Full Walkthrough of AI Autonomously Controlling a Mac Desktop

OpenAI's Codex is evolving from a coding assistant into a true "computer operator." The newly launched Computer Use feature enables Codex to control any application on a Mac just like a human—moving the mouse, clicking buttons, typing text—all without interrupting your workflow. In their latest demo, OpenAI engineers Romain and Ari showcased the full capabilities of this feature, from creating virtual machines to playing music on Spotify to sending iMessage texts. Codex's performance is impressive.

Computer Use has been one of the most closely watched capability directions in AI over the past two years. In October 2024, Anthropic was first to release Claude's Computer Use feature, allowing AI to control desktop applications through screenshot recognition and simulated mouse/keyboard operations. Google DeepMind's Project Mariner, Microsoft's UFO, and other projects quickly followed. The core significance of this capability is that it frees AI from being limited to text conversations or API calls—it can directly operate graphical user interfaces (GUIs) designed for humans, theoretically completing any task a human can do on a computer. Previous approaches universally suffered from two pain points: slow speed (relying on repetitive screenshot-recognize-act loops) and exclusivity (users couldn't use their computer while AI was working). OpenAI's approach specifically addresses both of these pain points.

Two Drag-and-Drop Steps to Set Up: Codex Computer Use Is Dead Simple

The onboarding experience for Computer Use is designed to be extremely smooth. On first use, the system presents a permission request window. After clicking "Allow," an animated transition guides you to System Settings, where you drag Codex into the Accessibility list. The entire setup requires just two drag-and-drop actions, after which Codex can begin controlling applications.

This minimalist onboarding design lowers the barrier to entry, allowing non-technical users to get started quickly. Once setup is complete, Codex enters Dock mode, standing by at all times.

Live Demo: AI Controlling Multiple Mac Apps Simultaneously

The demo showcased a very typical use case—creating a Mac virtual machine in UTM. UTM is an open-source virtualization tool based on QEMU, designed specifically for macOS, supporting various operating systems on Apple Silicon Macs. This task normally requires users to manually click through numerous settings, select OS images, configure CPU and memory parameters, and run the macOS installation assistant—a tedious and time-consuming process. With Codex, you simply type "make a new Mac VM in UTM," and it automatically launches UTM and completes all the steps.

Even more impressive is Codex's multitasking ability. While the virtual machine was downloading macOS, the presenter issued two additional tasks: play work-appropriate music on Spotify, and add an evening reminder in the Reminders app. Codex simultaneously drove three different applications, each running independently without interference.

Here's a critical technical highlight: Codex uses a virtual cursor that is independent from the user's cursor. This means while Codex operates applications in the background, you can continue using your own mouse for other things. In traditional Computer Use approaches (including Anthropic Claude and most RPA tools), AI simulates system-level mouse and keyboard events to manipulate the interface, meaning AI operations directly move the user's real cursor, making human and machine operations mutually exclusive. Codex's virtual cursor leverages the programmatic interaction capabilities within macOS's Accessibility API—using the AXUIElement interface to directly execute actions (such as clicks and confirmations) on target UI elements without actually moving the system cursor to the corresponding position. This approach essentially interacts with applications at the API level rather than at the input device level, allowing AI operations and user mouse/keyboard operations to proceed in parallel on completely independent channels. Ari specifically emphasized that this is the fundamental difference from every other Computer Use implementation on the market—all other approaches completely take over your computer, preventing you from using your device while AI is working.

Powered by the Spark Model: Achieving Superhuman Operation Speed

Computer Use supports not only multimodal models but also the faster Spark model. Spark is OpenAI's lightweight high-speed inference model, optimized for low-latency scenarios. Its inference speed is significantly faster than large multimodal models while maintaining high standards in text understanding and instruction following.

Behind this lies an important technical innovation: traditional Computer Use approaches rely entirely on screenshots to understand interfaces, while the Codex team deeply leverages macOS's Accessibility Framework to extract textual descriptions of the interface. The macOS Accessibility Framework was originally designed as accessibility infrastructure for visually impaired users. It requires every application to expose its UI elements (buttons, text fields, menus, sliders, etc.) to the system as a structured semantic tree (Accessibility Tree), containing each element's role, label, value, position coordinates, and executable actions. The screen reader VoiceOver works based on this framework. The Codex team cleverly repurposes this framework for AI interface understanding—compared to pure screenshot approaches, the Accessibility Tree provides precise structured data rather than pixel information that requires visual model parsing. This not only dramatically improves recognition accuracy but also eliminates the hard dependency on multimodal visual capabilities.

This approach brings two significant advantages:

• Higher accuracy: The model can "see" content beyond scrollable regions, gaining deeper understanding of each UI element's role and function, no longer limited to pixel information within the current visible screen area
• Faster speed: Since image input isn't mandatory, the non-multimodal Spark model can be used, skipping visual encoder processing overhead, compressing each operation's decision time from hundreds of milliseconds down to tens of milliseconds, achieving superhuman operation speed

In the demo, Ari used the Spark model to send an iMessage—opening the Messages app, finding a contact, typing text, and sending—all completed within seconds, far faster than manual human operation. Ari described the performance as "superhuman."

Security Mechanisms: Per-App Authorization Ensures Privacy Isolation

For an AI tool that can control an entire computer, security is naturally users' top concern. The Codex team has implemented a strict isolation strategy.

Current mainstream Computer Use solutions differ significantly in their security models. Anthropic's Claude Computer Use requires users to run a complete desktop environment in a Docker container or virtual machine, where AI can see and operate everything within that environment. Most RPA (Robotic Process Automation) tools require full screen recording and input control permissions. The common problem with these approaches is overly coarse permission granularity—it's either all-or-nothing.

Codex's Computer Use adopts a fundamentally different per-app authorization mechanism, borrowing the permission management philosophy from mobile operating systems (iOS/Android):

• Codex requests user permission when accessing any application for the first time
• Once authorized, Codex can only see and operate that specific application
• Unauthorized applications are completely invisible and non-interactive to Codex
• It doesn't stream the entire desktop or access all files

From a technical implementation perspective, this is possible because macOS's Accessibility API natively supports per-process UI element tree queries. Codex simply needs to filter out process IDs of unauthorized applications to achieve isolation, without requiring complex sandboxing mechanisms. This design lets users confidently authorize development tools and productivity apps to Codex while keeping applications containing sensitive information (such as banking clients, password managers, etc.) isolated. Compared to solutions requiring full desktop access, this fine-grained permission control establishes stronger user trust.

From Specialized Models to General Capabilities: The Technical Evolution of Computer Use

Ari revealed an important technical backstory: early Operator and ChatGPT Agent products used independently trained models specifically for Computer Use—a specialized model called CUA (Computer-Using Agent). These models were fine-tuned on top of standard GPT with extensive GUI interaction data to give them screenshot understanding and operation planning capabilities. However, maintaining independent specialized models means higher training costs, more complex deployment architectures, and disconnection from the main model's capability iterations.

Now, OpenAI's research team has integrated these capabilities into the mainline GPT models. This means Codex's Computer Use feature is built on the same models available through the API, and developers can leverage these capabilities to build their own applications.

This shift reflects an important trend in the AI industry—"capability convergence": various specialized skills (code generation, tool use, interface operation, multimodal understanding) no longer require separate models but exist as built-in capabilities of general foundation models. This not only simplifies OpenAI's internal workflows but also signals that Computer Use will become a standard feature of AI models rather than a separately trained specialized skill. Third-party developers calling standard models through the OpenAI API will be able to directly access Computer Use capabilities in the future without additional integration.

Future Outlook: 2x, 5x, 10x Human Operation Speed

Ari's vision for Computer Use is crystal clear: making AI operate computers at 2x, 5x, or even 10x human speed. When this goal is achieved, Computer Use will become indispensable—users will want to delegate nearly all routine computing tasks to AI, freeing themselves to focus on what truly matters.

Computer Use is currently live on Mac, with a Windows version under active development. Combined with Codex's existing file system access and online service plugin capabilities, Computer Use fills in the final piece of the puzzle—local application control—making Codex a truly all-capable digital assistant. From a broader perspective, the maturation of Computer Use may redefine the human-computer interaction paradigm. In the future, users won't need to learn how to operate each application; they'll simply describe their intent in natural language, and AI will complete all operations at the GUI level. This is essentially building a new interaction abstraction layer on top of the existing software ecosystem.

As Romain said at the end of the demo: "Try using it for your most complex tasks—the kind that requires switching between five apps and takes hours of work." This is perhaps the best way to put Computer Use to the test.

Key Takeaways

• Codex Computer Use enables AI to control any Mac application like a human, using an independent virtual cursor without interrupting the user's normal workflow
• Leverages the macOS Accessibility Framework to extract interface text information, combined with the Spark model to achieve superhuman application operation speed
• Employs a per-app authorization security mechanism where Codex can only access explicitly authorized apps, with unauthorized apps completely isolated
• Computer Use capabilities have migrated from specialized models to mainline GPT models, allowing developers to build similar features via API
• Supports parallel multi-app operation, simultaneously driving multiple applications to independently complete tasks