Computer-Using Agents: A New General-Purpose Paradigm for Embodied AI

In many areas of artificial intelligence, general-purpose models have gradually replaced hand-designed methods and specialized architectures. This trend is now spreading to embodied AI. A study from the ECCV workshop competitions proposes that embodied tasks can be reformulated as interaction with digital software, and that computer-using agents (CUAs)—leveraging skills acquired through software operation—can directly transfer to tasks in physical environments, demonstrating potential as "embodied generalists."

From Specialized to General: A Paradigm Shift in Embodied AI

Traditional embodied AI often relies on architectures and hand-crafted features designed for specific tasks, with limited generalization ability. The core claim of this study is that embodied tasks can essentially be viewed as interaction with digital software. Specifically, the researchers built a class of software that renders the agent's view of the physical environment on screen and provides a control interface for acting within the environment. In this way, a computer-using agent (CUA) can complete embodied tasks by processing screenshots and keyboard/mouse operations, exactly as it would operate ordinary software.

The cleverness of this approach lies in bypassing the need to separately design perception and motion control modules for embodied tasks, directly reusing the general capabilities CUAs have accumulated through software operation.

Computer-Using Agent (CUA) refers to an AI agent capable of operating graphical user interfaces (GUIs) like a human. Its typical working loop is: capture the current screen, have a vision-language model understand interface elements and task state, then output low-level actions such as mouse clicks, drags, or keyboard input, forming a closed-loop interaction with the environment. Representative works include web/desktop operation agents built on large models, which are typically trained on large amounts of software operation trajectories and thus possess reusable interface operation capabilities such as element localization, form filling, and multi-step planning.

Embodied AI, on the other hand, refers to an intelligent form in which agents complete tasks by interacting with physical or simulated environments through a body, covering scenarios such as navigation, manipulation, and grasping. Traditional methods mostly design perception modules and motion controllers separately for each task, with limited sample efficiency and generalization ability. The intersection of CUAs and embodied AI lies in the fact that both require mapping high-dimensional visual input to sequential action decisions; the only difference is whether actions act on screen coordinates or physical actuators.

Cross-Competition Validation: Generality Without Fine-Tuning

To validate this idea, the research team used a single CUA design, changing only the software interacted with, and participated in four ECCV workshop competitions, achieving first, second, third, and sixth places respectively, without any competition-specific agent tuning.

These results strongly indicate that the skills CUAs learn in software operation—such as visual localization, precise clicking, and exploration—can already transfer to embodied tasks. This gives CUAs a much broader range of applications in embodied AI than previously recognized.

ECCV workshop competitions refer to evaluation challenges held at various thematic workshops of the European Conference on Computer Vision (ECCV), mostly focusing on frontier subproblems such as embodied navigation, vision-language navigation, and object manipulation. Such competitions typically provide a unified simulation environment, task definitions, and evaluation protocols, and participants must submit models on unseen scenes and instructions. Because competitions across different workshops vary greatly in observation formats, action spaces, and task goals, a single model being able to participate in multiple competitions and achieve rankings is itself strong evidence of cross-task generalization ability.

Skill Transfer: From Software Operation to Physical Interaction

Why can software operation skills transfer to embodied tasks? The key is that both share similar interaction primitives: observing the screen (visual perception), moving the cursor (localization and navigation), clicking (action execution), and exploring the interface (trial-and-error learning). CUAs are trained on these primitives across large-scale software environments, developing robust visual localization and precise control capabilities. When the physical environment is mapped to a software interface on screen, these capabilities naturally apply.

The researchers argue that this transfer not only lowers the barrier to developing embodied AI but also suggests a path toward general embodied intelligence.

Interaction primitives refer to the minimal action-perception units that compose complex tasks, such as localizing a target, moving a pointer, executing a click, and exploring an interface. Cognitive science and robotics have long held that complex skills can be decomposed into combinations of shared primitives, and that transfer is possible because different tasks reuse the same set of primitives. CUAs, trained on massive trajectories in software environments, have precisely honed these primitives to be sufficiently robust; when the physical environment is encapsulated as an interface, the correspondence at the primitive level allows policies to be reused without relearning low-level control.

A Roadmap Toward General Embodied AI

Based on these findings, the authors advocate that computer-using agents can serve as embodied generalists. Furthermore, they propose that by following the training strategies that make computer-using agents general, embodied AI can likewise converge to a single general model, thereby ushering in a new era of general embodied AI.

This view aligns with the broader trend in AI from specialized models toward general models. If it holds, future embodied intelligence development may no longer require custom hardware or algorithms for each task, but instead achieve rapid adaptation through unified software interfaces and general agents.

The project website provides more details: https://pranjal2041.github.io/cua-embodied

One-line Summary

Research shows computer-using agents can transfer software operation skills to perform embodied tasks with general-purpose potential.

Paragraph Summary

The article introduces a study from the ECCV workshop competitions, arguing that embodied tasks can be reformulated as interaction with digital software: researchers built software that renders physical environment views on screen and provides control interfaces, enabling computer-using agents (CUAs) to complete embodied tasks via screenshots and keyboard/mouse operations, thereby bypassing the design of specialized perception and motion modules. The team used a single CUA design, changing only the software interacted with, and participated in four ECCV workshop competitions, placing first, second, third, and sixth, without competition-specific tuning. The article further analyzes the mechanism of skill transfer, noting that interaction primitives such as observation, localization, clicking, and exploration are shared between software and physical tasks, and advocates CUAs as embodied generalists, suggesting that following their general training strategies can lead embodied AI to converge to a single general model.