Deep Dive: CMU's Public Course on Computer Use Agents — Evolution, Evaluation, and Training

Computer Use Agents (CUAs) are rapidly evolving from a niche research topic into a core capability of frontier large models. In Lecture 7 of CMU's 2026 AI Agent public course, multimodal machine learning expert J.Y. Koh systematically traced the technical arc of CUAs — from toy environments nine years ago to autonomous operation that can now rival or even surpass human performance. This article breaks down how CUAs work, how they're evaluated, and how they're trained.

What Are Computer Use Agents

CUAs are fundamentally different from the SWE agents or pure-text agents covered earlier in the course. They receive graphical user interfaces (GUIs) as input — the same interfaces humans use every day. Whether it's web browsing, desktop interaction, or mobile apps, any agent that observes through screenshots and executes human-like actions — clicks, scrolls, swipes, and keystrokes — falls under the CUA umbrella.

They operate on an observe-reason-act loop: a user task plus the current screenshot is fed into a model, which produces a chain of thought and an action to execute. The environment executes the action, returns a new screenshot, and the cycle repeats until the task is complete or the budget runs out. Take "buy me a blue mug" as an example: the model identifies the blue mug on screen and clicks it, then scrolls to find shipping information, advancing step by step. Tasks typically conclude with a binary 0/1 reward, though some scenarios support fine-grained partial rewards.

This mechanism looks straightforward today, and modern models are quite capable — but that hasn't always been the case.

Chain of Thought (CoT) is a technique that prompts large models to output intermediate reasoning steps before arriving at a final answer, first systematically introduced by Google researchers Wei et al. in 2022. CoT is especially critical for CUAs: computer operation often requires multi-step planning — assessing the current page state, deciding the next action, and backtracking on failure. Explicit reasoning steps allow models to handle longer task dependency chains and make debugging and interpretation possible. Unlike pure-text tasks, CUA chain of thought must integrate visual information (e.g., "there's a blue button in the top-left corner") with operational intent ("I need to click it to reach the shopping cart"), placing higher demands on the collaborative visual understanding and language reasoning of multimodal models.

Nine Years of Evolution: From World of Bits to GPT-6 Astra

The first CUA paper was actually published by OpenAI back in 2017 under the project name World of Bits. It constructed a highly simplified synthetic environment trained with reinforcement learning reminiscent of early game agents. The project was remarkably ahead of its time, but all models performed poorly even on these simplified tasks.

As language models matured, Webshop in 2022 introduced a more realistic simulated shopping environment. In 2024, CMU Graham Lab's Web Arena became a landmark work — it populated environments with real website data scraped from Amazon, Reddit, and others, and models finally began to show meaningful progress. Released around the same time, OS World expanded the battleground from the web to full desktop operation, supporting Linux, Windows, and a range of applications, making it one of today's most standard CUA benchmarks.

Entering 2025 and 2026, one notable shift has been the maturation of coding agents. Earlier benchmarks relied on manually written open-source applications populated with real data, while newer benchmarks like My PC Bench use coding agents to end-to-end generate entire applications and websites — for example, a fully LLM-generated Gmail clone with entirely synthetic data. These synthetic environments are sometimes even cleaner and less noisy than real applications.

The lecture also showcased a GPT-6 Astra demo released this month: it operates CAD software for 3D modeling in a video sped up 30×. This means the actual runtime is still 30× slower and costly, but in terms of capability it has "largely solved" most computer operation benchmarks, surpassing the human median. In nine years, CUAs have gone from toy environments to expert-level autonomous operation.

Four Evaluation Paradigms and Their Inherent Challenges

Compared to software engineering or mathematics, computer operation is a task that's "harder to verify" — even humans may not agree on what constitutes successful completion, making programmatic verifiers especially tricky to write. The lecture categorizes evaluation approaches into several types.

Static Evaluation: Fast but Distorted

Static evaluation measures whether each action step in a trajectory is correct. Take ScreenSpot Pro as an example — it's a grounding benchmark with roughly 1,500 screenshot samples that awards a point whenever the model's predicted click coordinate lands within the target bounding box. Mind2Web tests richer action imitation, predicting the next action given a screenshot and task. This type of evaluation requires no real environment, making it extremely fast and cheap, but its biggest flaw is severe false negatives — a given task often has multiple valid action paths, while static annotations only accept one canonical set of answers.

Grounding in the CUA context specifically refers to a model's ability to map natural language descriptions to concrete pixel coordinates on screen — for example, given the instruction "click the 'Add to Cart' button," the model must output the button's precise location. This capability is foundational to all subsequent operations: if grounding fails, any click action will inevitably be wrong. Benchmarks like ScreenSpot Pro isolate and evaluate this capability independently, because in end-to-end evaluation it's often hard to attribute task failure: was it that the model didn't understand the task, couldn't locate the target element, or made errors in action sequence planning? Standalone grounding benchmarks can more precisely diagnose a model's visual perception shortcomings and facilitate targeted pre-training data collection.

End-to-End Evaluation: Closer to the Real Goal

End-to-end evaluation truly measures whether the final outcome was achieved. In simulated environments, agents run freely, and after the task concludes, unit-test-style programmatic checks are run — for example: is the blue mug in the shopping cart, is the price under $20, is the order count zero? Web Arena pioneered this paradigm, containing roughly 800 tasks across four applications with all validators hand-written by CMU graduate students. The downside is the potential for reward hacking — agents may find ways to satisfy the validator without genuinely completing the task. Visual Web Arena further upgraded this to a multimodal problem, demonstrating the value of the screenshot-based approach.

LLM-as-Judge: Flexible but Gameable

When writing a validator for every task isn't feasible, another large model can serve as judge. Web Voyager was one of the first benchmarks to test CUAs on real web pages, covering 15 websites and 600 tasks; Online Mind2Web further scaled to 136 websites and 300 tasks, with its judge achieving roughly 85% agreement with humans. However, evaluating on real web pages brings complications — heavy agent traffic frustrates website owners — so this style of evaluation has gradually declined, and modern models have largely mastered these benchmarks anyway.

Human Evaluation: The Expensive Gold Standard

Both programmatic validators and LLM judges are ultimately proxy metrics. Human evaluation remains the gold standard for judging whether an agent succeeded, but its high cost means it's typically done only once or twice to validate benchmark reasonableness.

Long-Horizon Tasks and Training Paradigms

As standard benchmarks have been conquered by modern models, the community has shifted toward long-horizon computer operation — tasks that might take humans hours to complete. Because final rewards are extremely sparse, long-horizon evaluation broadly adopts rubric-style fine-grained rewards: both state checks that verify intermediate sub-tasks and end-to-end process rewards. CMU's Odyssey benchmark contains 200 tasks, each with multiple rubrics (e.g., "plan a Palm Springs wedding itinerary" must satisfy 8 criteria), enabling partial scores rather than a single binary judgment.

OS World 2 challenges hour-scale workflows, with 108 tasks whose median completion time for humans is 1.6 hours, involving professional software like CAD modeling and photo editing; frontier agents average roughly 300 calls per task. CMU's CUA World Long similarly targets professional long-tail scenarios in healthcare, engineering, and architectural design. The lecture noted that early models scored extremely low on these benchmarks, while recent models like GPT-6 can now reach close to 90% — in a sense, general-purpose computer operation is approaching being "solved."

How Models Are Represented

A CUA is fundamentally a Vision-Language Model (VLM). Naturally-described goals and screenshot inputs feed into a policy network, and the model produces structured actions in text form (e.g., a click with xy coordinates). After execution, a new screenshot is returned, concatenated with history and fed back in, forming an ever-growing interleaved image-text action history. Text is converted to token embeddings, screenshots to visual embeddings, and the interleaved sequence is fed into a Transformer — a representation that is simple yet surprisingly effective.

The lecture specifically addressed the question of why screenshots rather than HTML: HTML is too messy and hard to obtain in desktop scenarios, while screenshots are universal across any device. Coordinates use a normalized 0–1000 representation that transfers reasonably well across resolutions and devices.

Three-Stage Training Pipeline

Mainstream CUAs build on pre-trained VLMs and train in three stages:

• Pre-training: Ingesting large-scale, noisier grounding data (screenshots + bounding boxes + coordinates) and action prediction data, extractable at web scale via JavaScript scripts. This teaches the model basic localization and screenshot comprehension.

• Supervised Fine-Tuning (SFT): Behavioral cloning on human demonstration data. Momo Webmix collected 36,000 human trajectories; AgentNet and others provide open-source data. Today, large amounts of synthetic data are also mixed in — one interesting approach gives a "teacher" model privileged information (such as an accessible accessibility tree) so it performs better, collects those trajectories, then distills into a student model that uses only screenshots. The lecture draws an analogy to Tesla training with LiDAR but inferring from pixel data alone.

• Reinforcement Learning (RL): Deploying the model into simulated environments to act autonomously, updating the policy based on rewards from rubrics, programmatic validators, or LLM judges. Data shows that post-SFT success rates of around 0.5 can climb above 0.6 after RL. RL is especially critical for long-horizon capabilities, since humans can barely provide demonstrations for extremely long tasks. Simulated environments are necessary because in the real world, repeatedly ordering Ubers would result in account bans or depleted budgets.

Accessibility Tree (A11y Tree) is the structured UI information exposed by operating systems and browsers for assistive technologies like screen readers. It contains the type, text label, coordinates, and interactive state of each interface element. Compared to raw screenshots, it provides a near-perfect "cheat sheet": instead of recognizing pixels, the model can directly read a button's semantic label for precise operation. However, in real deployments, native desktop applications often lack proper accessibility support, and the accessibility tree may be unstable across app versions or system configurations. The Tesla analogy in the lecture maps directly to this idea: training with LiDAR for precise 3D information (analogous to the accessibility tree), then inferring with only camera pixels (analogous to screenshots), using knowledge distillation to help the vision model internalize the understanding implicit in structured information while remaining compatible with pure visual input.

Long-horizon tasks are one of the core challenges in AI agent research — tasks that require completing dozens or even hundreds of interdependent steps in sequence to achieve a goal. The difficulty comes from two main sources: first, reward sparsity — feedback only arrives when the task is fully complete, and all intermediate steps provide almost no training signal for policy updates, making traditional RL extremely difficult to train effectively; second, error accumulation — minor deviations in early steps amplify along the sequence, causing all subsequent steps to fail. Rubric-style rewards address this by decomposing long-horizon tasks into independently verifiable sub-goals, providing feedback signals for each intermediate state — essentially converting a sparse reward problem into a denser reward form, and currently one of the most practical engineering solutions for mitigating the difficulties of long-horizon task training.

Open Problems

J.Y. Koh concluded by noting that despite impressive capabilities, CUAs still face several major challenges: speed remains a bottleneck (GPT-6 Astra runs 30× slower than its demo suggests), personalization is a research direction still waiting to be developed, infrastructure and user experience have numerous pain points, and multi-agent systems hold significant imaginative potential. Computer use agents have come a remarkable distance from toy environments nine years ago, but seamlessly integrating into everyday workflows remains a work in progress.