OSWorld 2.0 Benchmark: Top AI Agent Achieves Only 20% Completion Rate — Long-Horizon Tasks Are the Achilles' Heel

The Bubble of High Scores on Short Tasks Has Burst

Today's AI Agents can browse the web, manage files, and even write code. But how well they actually perform on real, complex work has long lacked an authoritative measurement standard. The OSWorld 2.0 benchmark, newly released by the XLANG Lab, was created precisely to fill this evaluation gap.

The quality of a benchmark determines whether our understanding of AI capabilities is accurate. Historically, the successive "saturation" of benchmarks like ImageNet, GLUE, and SuperGLUE drove wave after wave of breakthroughs in deep learning, but also exposed a common problem: models tend to overfit to benchmarks, performing brilliantly on test sets while falling flat in real-world scenarios. Take ImageNet as an example — top models had pushed error rates below human levels by around 2017, but researchers subsequently found that these models suffered accuracy drops of 20–30 percentage points on slightly distribution-shifted datasets (such as ObjectNet). They had learned statistical patterns in the dataset, not genuine visual understanding. GLUE and SuperGLUE were similarly "conquered" within just a few years, forcing researchers to continually design tasks closer to real understanding.

GLUE (General Language Understanding Evaluation) and SuperGLUE are the most representative comprehensive evaluation suites in the field of natural language processing. GLUE was released in 2018 and included 9 tasks such as sentiment analysis, textual entailment, and question answering. Because top models surpassed the human baseline within just one year, the more difficult SuperGLUE was introduced in 2019. However, SuperGLUE was likewise "conquered" by models around 2021. This recurring cycle reveals a deep dilemma in language understanding benchmarks: any fixed test set, once subjected to large-scale targeted optimization by the research community, gradually loses its ability to distinguish between "genuine understanding" and "pattern memorization." For this reason, the field has begun shifting toward dynamic benchmarks (such as Dynabench, which allows researchers to continuously add new samples that models tend to get wrong) and capability evaluation frameworks centered on task completion. The OSWorld series is a concrete embodiment of this trend in the domain of computer operation.

Notably, when model scores on a benchmark approach or exceed human levels, the research community often misinterprets this as "this capability has been solved" rather than "this evaluation tool has become obsolete." This cognitive bias can systematically misdirect resource allocation and research priorities — the designers of OSWorld 2.0 clearly drew lessons from this history, choosing to raise the difficulty threshold simultaneously along three dimensions (task duration, application breadth, and scoring granularity) rather than simply increasing the number of tasks. This historical cycle is replaying in the AI Agent space. This phenomenon is especially pronounced in the AI Agent domain — if desktop operation benchmarks include only quick, simple tasks, models can score high by memorizing a limited set of operation patterns without needing any real "workflow planning" capability.

The earlier OSWorld 1.0 was a product of exactly this evaluation limitation, creating an illusion that "desktop operation has been solved" — Claude Opus achieved 83.5% accuracy on 1.0. But a closer analysis reveals that those tasks were mostly quick, simple operations involving one or two applications at most, rewarding only independently completed small steps without requiring coherent long-horizon workflows. This high accuracy rate seriously overestimated AI's ability to complete real work end-to-end.

OSWorld 2.0 operates at an entirely different scale: it contains 108 long-horizon real computer tasks, with a median completion time of 1.6 hours for skilled humans — approximately 48 times that of version 1.0 (2 minutes). Mainstream AI agents require an average of over 300 steps to complete each task, compared to about 30 steps for version 1.0. The sheer magnitude of this gap alone reveals the chasm in difficulty between the two benchmarks.

Three Core Design Principles and Real Environment Setup

OSWorld 2.0's design follows three core principles:

• Real workflows: All tasks come from real professional scenarios, sourced through expert interviews and professional annotator research. Input materials are also adapted from real materials rather than synthetically generated.
• Long-horizon structure: Task difficulty stems from interdependent steps across multiple applications, not simple repetition or stitching together of unrelated subtasks.
• Diverse challenge dimensions: Each task is annotated with 10 capability testing dimensions, covering cross-source reasoning, implicit state inference, conflict resolution, dynamic environment adaptation, and more.

To achieve these goals, the team built an exceptionally comprehensive testing environment. In an AI research landscape where the reproducibility crisis is receiving increasing attention, the engineering decision to self-host 31 task-related web services carries significant methodological importance. The reproducibility crisis in AI research has drawn widespread concern in recent years — numerous experimental results cannot be reproduced due to missing original code, dependence on specific hardware, or changes to external services. Real internet environments present three major evaluation obstacles: frequent website UI updates that invalidate screenshot recognition, anti-scraping mechanisms that interfere with automated testing, and the inability to reset historical states, making experiments non-repeatable. By building their own email services, banking portals, team chat tools, and more, the research team achieved complete control over the test environment, allowing any researcher to reproduce experiments from identical initial states. This philosophy of treating "evaluation infrastructure" as a research artifact equally important as "evaluation tasks themselves" is foundational to building a trustworthy knowledge accumulation system. Desktop applications were also expanded from 1.0's LibreOffice, GIMP, and VS Code to professional software including Slack, LinkedIn, Reaper, WPS, GitLab, Overleaf, Zotero, and others.

Each task features a coherent user profile preset as the initial state. The workspace may include local files, open documents, browser tabs, account records, message history, and more. These materials are designed around a unified profile, with identifiers, dates, and amounts across different sources kept consistent.

Simulated Users and Dynamic Environments

Two clever designs in the testing deserve attention. The first is simulated users: some tasks contain missing evidence or ambiguous constraints, requiring the AI to proactively ask for clarification through user channels. The simulator returns only pre-configured, limited information — this tests whether the AI "knows when it should ask." The second is dynamic environments: self-hosted services inject new emails or chat messages during task execution, testing whether the AI can continuously monitor channels, identify new constraints, and revise prior decisions, rather than treating the initial state as the final state and executing straight through.

Notably, these two designs touch on the core challenges of AI Agent architecture. Current mainstream computer operation Agents rely on Vision-Language Models (VLMs) to understand screen states, then execute specific operations through tool calls. Their structure consists of three layers: perception, planning, and execution. The perception layer converts screenshots into structured environment descriptions; the planning layer generates action sequences based on the current state and task objectives, typically through Chain-of-Thought reasoning; the execution layer maps planning outputs to specific mouse clicks, keyboard inputs, or API calls. This architecture performs adequately on short tasks, but its fundamental limitation lies in the "one-shot" nature of the planning layer — most Agents generate a plan at the start of a task and then mechanically execute it, lacking mechanisms to perceive and respond to mid-course state changes. The dynamic environment design that injects new information directly tests the Agent planning layer's ability to respond when "initial assumptions become invalid" — something extremely common in real workplace scenarios but almost never systematically evaluated in previous benchmarks.

A More Refined Scoring Mechanism and Three-Layer Quality Audit

The scoring mechanism received a major upgrade. Version 1.0 used binary pass/fail scoring, which is too coarse for long-horizon tasks — "zero progress" and "only one step short" receive exactly the same score. Version 2.0 adopts fine-grained checkpoint partial credit, with an average of 27.25 checkpoints per task. It does not require a fixed order and supports multiple valid solution paths.

This design is highly aligned in concept with "Dense Reward" in reinforcement learning. Traditional binary scoring is equivalent to "sparse reward" — positive signals are given only when the task is fully completed, making it difficult for models to learn valuable intermediate information from failures. Sparse reward has been one of the core challenges in reinforcement learning for decades: the famous Montezuma's Revenge game was a landmark hard problem — early algorithms could barely learn effectively because rewards were only given upon reaching the end. Robotics similarly suffered, with robotic arm grasping success rates near zero under sparse rewards until Curriculum Learning and Reward Shaping techniques emerged.

Reward Shaping is one of the classic technical approaches to solving sparse rewards. Its core idea is to introduce additional intermediate signals beyond the original reward, guiding the agent to explore in the right direction. However, Reward Shaping carries an inherent risk: poorly designed intermediate rewards can lead agents to take shortcuts — finding ways to earn rewards without truly completing the intended objective. This is known as "Reward Hacking" (a manifestation of Goodhart's Law). OSWorld 2.0's checkpoint mechanism, with its "no fixed order" design, avoids the path dependency problem in traditional curriculum learning while using rigorous reward hacking audits to close specification gaming loopholes. The team specifically checked whether each checkpoint could be triggered without genuinely completing the corresponding sub-goal — this methodological rigor is a crucial guarantee of the benchmark's credibility. The checkpoint mechanism decomposes a long task into an average of 27.25 independently verifiable milestones, providing models with meaningful progress signals while also offering a more refined foundation for constructing reward functions for future Agent training.

The checkpoint mechanism of OSWorld 2.0 is essentially a manually designed Reward Shaping strategy, but its "no fixed order" design avoids the path dependency problem in traditional curriculum learning, providing Agents with more flexible exploration space. This design also holds significant value for future Agent training: researchers can build more refined reward functions based on these checkpoints, training Agents that truly understand "task progress" rather than focusing solely on "final results."

Evaluation prioritizes functional verification (directly checking environment states and output artifacts), using model-based evaluation only as a limited supplement when open-ended judgment is needed. Overall, model evaluation accounts for only 11.53% of the total score, and no task has model evaluation exceeding 50% — maximizing objectivity.

Quality assurance follows a three-layer process: first, AI generates initial unit tests; then two independent annotators complete the task from start to finish and cross-check each other; finally, multiple frontier AI agents run through the tasks to expose scoring vulnerabilities. The team also specifically audited two opposing risks — reward hacking (AI receives credit without meeting the real requirement, e.g., a task requires walking directions but evaluation only checks waypoints, so driving directions also score) and false negatives (correct solutions wrongly judged as incorrect due to format differences).

Test Results: Even the Strongest Model Achieves Only 20.6% Completion

The research team tested 7 mainstream computer operation model families, including Claude Opus 4.8/4.7, GPT-5.5, and leading open-source models, with step budgets set at 150, 300, and 500.

Under the 500-step budget with batch action mode, the key results are sobering:

• Claude Opus 4.8 (with maximum thinking depth + batch tool calls): completion rate of only 20.6%, partial score of 54.8%, average cost per task approximately $72.4;
• Claude Opus 4.7: 18.2% completion rate, approximately $33.6 per task;
• GPT-5.5: 13.0% completion rate, only $25.5 per task (less than one-fifth of Opus), achieving the highest token efficiency;
• The best-performing open-source models, MiniMax M3 and Kimi, achieved only 4.6% completion rate.

The data shows that pursuing higher completion rates means steep increases in token costs. More critically, completion rate is extremely sensitive to task length: on the longest workflows, AI completion rates drop to nearly 0, even though partial scores can remain relatively high. Many tasks that humans find simple are surprisingly difficult for AI — brute-force scaling by simply increasing the step budget has very limited effectiveness here.

Root Cause of Failure: Not Inability, but "Forgetting"

These failures don't stem from AI lacking basic GUI operation or code-writing abilities. Research found that AI performs reasonably well on local actions; the core bottleneck is task state maintenance over long horizons.

The technical root cause of this problem is worth understanding in depth. While current mainstream large models support context windows of hundreds of thousands or even millions of tokens, research shows that models pay significantly more attention to information at the head and tail of the context than to the middle — a phenomenon called "Lost in the Middle," systematically documented by Stanford University's Nelson Liu et al. in 2023. Mechanistically, this is closely related to the Transformer's attention distribution and positional encoding methods — during pretraining, models encounter far fewer long-text samples than short-text samples, resulting in insufficient training for processing information in the middle of long sequences. This finding directly explains the OSWorld 2.0 evaluation results: in task executions exceeding 300 steps, user constraints recorded at step 50 are very likely to be "forgotten" by step 200 — not because the context window isn't large enough, but because the attention mechanism cannot effectively retrieve key information over long distances.

Working Memory theory from cognitive science provides a useful lens for understanding this problem — cognitive scientist Alan Baddeley's working memory model divides the human short-term information processing system into three components: the phonological loop, the visuospatial sketchpad, and the central executive. Its core characteristic is limited capacity that can be extended through external tools. Humans overcome working memory capacity limitations in complex tasks through external tools like sticky notes, checklists, and calendars. This directly inspires the future direction of AI Agent architecture: the need for similar "external working memory" modules that store task state in a structured manner outside the model's context, with on-demand retrieval and updates. In task executions exceeding 300 steps, constraints raised by users early on, content from new emails appearing mid-task, and the states of previously completed steps all need to be continuously tracked and referenced by the model throughout execution. This goes beyond a simple "memory capacity" issue — it requires a structured working memory mechanism.

Typical model mistakes include: omitting constraints explicitly stated by users; ignoring new information that appears mid-task; guessing directly when uncertain rather than proactively asking; and skipping necessary verification steps. Even more noteworthy, throughout the entire task execution process, less than 7% of steps are devoted to detecting and fixing the model's own errors.

These weaknesses are concentrated in scenarios requiring recovery and maintenance of hidden states, such as implicit state inference, multi-item state tracking, conflict resolution, and dynamic environment adaptation. By task label distribution, the most common challenges are cross-source reasoning (42.6%), visual-spatial precision (41.7%), with implicit state inference and multi-item state tracking each at 39.8% — precisely the weakest areas for current AI. Solutions currently being explored in academia include structured state databases (categorized storage of task constraints, completed steps, and items pending verification), active summarization mechanisms (periodic compression and extraction of key information from context), and memory-augmented architectures (such as MemGPT, which brings operating system virtual memory management concepts to LLM context management). However, no mature solution has yet proven stable in real long-horizon scenarios. The OSWorld 2.0 finding that only 7% of steps are used for self-correction provides a clear quantitative baseline for this research direction.

Direction Matters More Than Parameters

The most fundamental value of OSWorld 2.0 lies in elevating the evaluation of AI computer operation from short tasks to real long-horizon work scenarios, bursting the bubble of high accuracy on short tasks and giving the industry a clear view of current AI Agent capabilities.

It also clearly points to future R&D directions: not simply stacking model parameters or increasing step budgets, but improving AI's long-horizon state maintenance, error self-detection and repair, dynamic environment adaptation, and proactive interaction capabilities. Task economic value mapping shows these workflows cover real high-value domains including document preparation (23.8%), software and databases (17.8%), and financial operations analysis (16.3%). Only once these challenges are overcome can AI truly take on complex work at the half-day scale.

OSWorld 2.0 is now fully open-sourced, including the environment, 108 tasks, self-hosted websites, and AI execution trajectories, providing a solid foundation for future research. We still have a long way to go before AI agents can truly "take over half a day's work."