Computer-use agents
AI agents capable of operating computer interfaces to complete tasks, typically based on vision-language models that perceive the screen through screenshots and execute mouse and keyboard actions.
Timeline (last 90 days)
Computer-using agents (CUAs) have demonstrated performance surpassing humans on multiple long-horizon benchmarks, but high computational cost limits their application in long-horizon tasks and real-time scenarios.
In CUAs, each screenshot is represented as thousands of visual tokens, which accumulate with the interaction history to form an extremely long context, and most visual tokens are either semantically irrelevant to the next action or temporally redundant across consecutive screenshots.
Through sparse visual representations, VTR makes the deployment of computer-using agents more feasible in long-horizon tasks and real-time domains such as video games.
For computer-use agents, latency is the core metric that determines practicality.
Common approaches to optimizing computer-use agent latency include compressing screenshot resolution to reduce token consumption, using local lightweight models to handle simple steps, and caching and reusing action flows.
All Facts (5)
Computer-using agents (CUAs) have demonstrated performance surpassing humans on multiple long-horizon benchmarks, but high computational cost limits their application in long-horizon tasks and real-time scenarios.
50%UnverifiedIn CUAs, each screenshot is represented as thousands of visual tokens, which accumulate with the interaction history to form an extremely long context, and most visual tokens are either semantically irrelevant to the next action or temporally redundant across consecutive screenshots.
50%UnverifiedThrough sparse visual representations, VTR makes the deployment of computer-using agents more feasible in long-horizon tasks and real-time domains such as video games.
50%UnverifiedFor computer-use agents, latency is the core metric that determines practicality.
50%UnverifiedCommon approaches to optimizing computer-use agent latency include compressing screenshot resolution to reduce token consumption, using local lightweight models to handle simple steps, and caching and reusing action flows.
50%