OSWorld 2.0 Benchmark: Top AI Agents Achieve Only 20% Completion Rate — Long-Horizon Tasks Are the Fatal Weakness

The Bubble of High Scores on Short Tasks Has Burst

Today's AI Agents can already help us browse the web, manage files, and even write code — but how well they actually perform on real, complex work has long lacked an authoritative measure. The OSWorld 2.0 benchmark, newly released by the XLANG Lab, was created precisely to fill this evaluation gap.

The quality of a benchmark determines whether our understanding of AI capabilities is accurate. Historically, the successive "saturation" of benchmarks like ImageNet, GLUE, and SuperGLUE drove wave after wave of deep learning breakthroughs, but also exposed a common problem: models tend to overfit to benchmarks, performing brilliantly on test sets while falling flat in real-world scenarios. Take ImageNet as an example — top models had pushed error rates below human level by around 2017, but researchers later discovered that these models suffered accuracy drops of 20–30 percentage points on slightly distribution-shifted datasets (such as ObjectNet). They had learned the statistical patterns of the dataset, not genuine visual understanding. GLUE and SuperGLUE were similarly "conquered" within just a few years, forcing researchers to continuously design tasks that better approximate real understanding.

GLUE (General Language Understanding Evaluation) and SuperGLUE are the most representative comprehensive evaluation suites in the NLP field. GLUE was released in 2018 and includes 9 tasks covering sentiment analysis, textual entailment, question answering, and more. Since top models surpassed human baselines within just one year, the more difficult SuperGLUE was introduced in 2019. However, SuperGLUE was similarly "conquered" by models around 2021. This recurring cycle reveals a deep dilemma in language understanding benchmarks: any fixed test set, once subjected to large-scale targeted optimization by the research community, gradually loses its ability to distinguish "true understanding" from "pattern memorization." This is why the field has begun shifting toward dynamic benchmarks (such as Dynabench, which allows researchers to continuously add new samples that models easily get wrong) and capability evaluation frameworks centered on task completion. The OSWorld series is a concrete embodiment of this trend in the domain of computer operation.

Notably, when model scores on a given benchmark approach or surpass human-level performance, the research community often misinterprets this as "this capability has been solved" rather than "this evaluation tool has become ineffective." This cognitive bias systematically misdirects resource allocation and research priorities — the designers of OSWorld 2.0 clearly drew lessons from this history, choosing to simultaneously raise the difficulty threshold across three dimensions — task duration, application breadth, and scoring granularity — rather than merely increasing the number of tasks. This historical cycle is repeating itself in the AI Agent domain. This phenomenon is especially pronounced in the AI Agent space: if desktop operation benchmarks only include quick, simple tasks, models need only memorize a limited set of operation patterns to score high, without needing genuine "workflow planning" ability.

The predecessor OSWorld 1.0 was a product of exactly this evaluation limitation, creating the illusion that "desktop operation has been solved" — Claude Opus achieved an 83.5% accuracy rate on version 1.0. But a closer look reveals that those tasks were mostly quick, simple operations involving at most one or two applications, rewarding only independently completed small steps without requiring coherent long-horizon work. This high accuracy rate severely overestimated AI's ability to complete real work end-to-end.

OSWorld 2.0 operates on an entirely different scale: it contains 108 long-horizon real computer tasks, with a median completion time for skilled humans of 1.6 hours — roughly 48 times that of version 1.0 (2 minutes). Mainstream AI agents require an average of over 300 steps per task, compared to about 30 steps in version 1.0. This scale difference alone reveals the chasm in difficulty between the two benchmarks.

Three Design Principles and Real-World Environment Setup

OSWorld 2.0's design follows three core principles:

• Real workflows: All tasks originate from real professional scenarios, gathered through expert interviews and professional annotator research. Input materials are adapted from real materials, not synthetically generated.
• Long-horizon structure: Task difficulty comes from interdependent steps spanning multiple applications, not from simple repetition or concatenation of unrelated subtasks.
• Diverse challenge dimensions: Each task is annotated across 10 capability testing dimensions, covering cross-source reasoning, implicit state inference, conflict resolution, dynamic environment adaptation, and more.

To achieve these goals, the team built an extremely thorough testing environment. In an AI research landscape where the reproducibility crisis is receiving increasing attention, the engineering choice to self-host 31 task-related web services carries significant methodological importance. The reproducibility crisis in AI research has attracted widespread concern in recent years — a large number of experimental results cannot be reproduced due to missing source code, dependence on specific hardware, or changes to external services. Real internet environments pose three major evaluation obstacles: frequent website UI updates that invalidate screenshot recognition, anti-scraping mechanisms that interfere with automated testing, and the inability to reset historical states that prevents experiment repetition. By building their own email services, banking portals, team chat tools, and more, the research team achieved full control over the test environment, allowing any researcher to reproduce experiments from the same initial state. This philosophy of treating "evaluation infrastructure" as a research artifact equally important as "the evaluation tasks themselves" is foundational to building a trustworthy knowledge accumulation system. Desktop applications were also expanded from the 1.0 set of LibreOffice, GIMP, VS Code, and others to include professional software such as Slack, LinkedIn, Reaper, WPS, GitLab, Overleaf, and Zotero.

Each task comes with a coherent user profile as the initial state. The workspace may include local files, open documents, browser tabs, account records, message history, and more. These materials are designed around a unified profile, with identifiers, dates, and amounts kept consistent across different sources.

Simulated Users and Dynamic Environments

Two clever designs in the testing are worth highlighting. The first is simulated users: some tasks involve missing evidence or ambiguous constraints, and the AI must proactively ask for clarification through user channels. The simulator only returns limited pre-configured information — this tests the AI's ability to "know when to ask." The second is dynamic environments: self-hosted services inject new emails or chat messages during task execution, testing whether the AI can continuously monitor channels, identify new constraints, and revise previous decisions, rather than treating the initial state as the final state and executing straight through.

Notably, these two designs strike at the core architectural challenges of AI Agents. Current mainstream computer operation agents rely on Vision-Language Models (VLMs) to understand screen states, then execute specific operations through tool calls. Their structure consists of three layers: perception, planning, and execution. The perception layer converts screenshots into structured environment descriptions; the planning layer generates action sequences based on current state and task objectives, typically through Chain-of-Thought reasoning; the execution layer maps planning outputs to specific mouse clicks, keyboard inputs, or API calls. This architecture performs adequately on short tasks, but its fundamental limitation lies in the "one-shot" nature of the planning layer — most agents generate a plan at the start of a task and then execute mechanically, lacking mechanisms to perceive and respond to mid-task state changes. The dynamic environment design that injects new information directly tests the planning layer's ability to respond when "initial assumptions become invalid" — something extremely common in real workplace tasks but almost never systematically evaluated in previous benchmarks.

A More Nuanced Scoring Mechanism and Three-Layer Quality Audit

The scoring mechanism has also been significantly upgraded. Version 1.0 used binary pass/fail scoring, which is far too coarse for long-horizon tasks — "zero progress" and "just one step short" receive exactly the same score. Version 2.0 adopts fine-grained checkpoint partial credit, with an average of 27.25 checkpoints per task. These don't require a fixed order and support multiple valid solution paths.

This design is highly aligned with the concept of "Dense Reward" in reinforcement learning. Traditional binary scoring is equivalent to "sparse reward" — positive signals are given only upon complete task success, making it difficult for models to learn valuable intermediate information from failures. Sparse reward has been one of the core challenges in reinforcement learning for decades: the famous Montezuma's Revenge game was long its signature hard problem, as early algorithms could barely learn effectively when rewards only came at the end; the robotics field suffered similarly, with robotic arm grasping success rates near zero under sparse rewards until techniques like Curriculum Learning and Reward Shaping emerged.

Reward Shaping is one of the classic technical approaches to solving sparse rewards. Its core idea is to introduce additional intermediate signals beyond the original reward to guide agents toward correct exploration. However, Reward Shaping carries an inherent risk: poorly designed intermediate rewards can cause agents to take shortcuts — finding ways to earn rewards without truly completing the intended objective. This is known as "Reward Hacking" or "specification gaming" (a manifestation of Goodhart's Law). OSWorld 2.0's checkpoint mechanism, with its "non-fixed order" design, avoids the path dependency problems of traditional curriculum learning while plugging specification gaming loopholes through rigorous reward hacking audits — the team specifically checked whether each checkpoint could be triggered without genuinely completing the corresponding sub-goal. This methodological rigor is a crucial guarantee of the benchmark's credibility. The checkpoint mechanism decomposes a long task into an average of 27.25 independently verifiable milestones, providing models with meaningful progress signals while also laying a more refined foundation for reward function construction in future agent training.

OSWorld 2.0's checkpoint mechanism is essentially a manually designed Reward Shaping strategy, but its "non-fixed order" design avoids the path dependency issues of traditional curriculum learning, giving agents more flexible exploration space. This design also holds significant value for future agent training: researchers can build more refined reward functions based on these checkpoints to train agents that truly understand "task progress" rather than focusing solely on "final results."

The evaluation prioritizes functional verification (directly checking environment states and output artifacts), using model-based evaluation only as a limited supplement when open-ended judgment is needed. Overall, model evaluation accounts for only 11.53% of total scores, and no task has model evaluation exceeding 50%, maximizing objectivity.

Quality assurance follows a three-layer process: first, AI generates initial unit tests; then two independent annotators complete the full task from scratch and cross-check; finally, multiple frontier AI agents run through the tasks to expose scoring vulnerabilities. The team also specifically audited two opposite risks — reward hacking (the AI receives credit without meeting the real requirement, such as a task requiring walking directions but the evaluation only checks waypoints, allowing driving directions to pass) and false negatives (correct solutions incorrectly marked as wrong due to format differences).

Test Results: Even the Strongest Model Achieves Only 20.6% Completion

The research team tested 7 mainstream computer operation model families, including Claude Opus 4.8/4.7, GPT-5.5, and leading open-source models, with step budgets set at 150, 300, and 500.

Under the 500-step budget with batch action mode, the key results are sobering:

• Claude Opus 4.8 (with maximum thinking depth + batch tool calls): completion rate of only 20.6%, partial score 54.8%, average cost per task approximately $72.4;
• Claude Opus 4.7: completion rate 18.2%, cost approximately $33.6;
• GPT-5.5: completion rate 13.0%, cost only $25.5 (less than one-fifth of Opus), highest token efficiency;
• The best-performing open-source models, MiniMax M3 and Kimi, achieved only 4.6% completion.

The data shows that pursuing higher completion rates means steep increases in token costs. More critically, completion rate is extremely sensitive to task length: on the longest workflows, AI completion rates drop to nearly 0, even though partial scores can remain relatively high. Many tasks that humans find easy turn out to be difficult for AI — brute-force scaling by simply increasing step budgets has very limited effect here.

Root Cause of Failure: Not Inability, but "Inability to Remember"

These failures don't stem from AI lacking basic GUI operation or code-writing abilities. Research found that AI performs reasonably well on local actions; the core bottleneck is maintaining task state across long horizons.

The technical root cause of this problem deserves deeper understanding. Although current mainstream large models support context windows of hundreds of thousands or even millions of tokens, research shows that models pay significantly more attention to information at the beginning and end of the context than in the middle — this is known as the "Lost in the Middle" phenomenon, systematically documented by Stanford University's Nelson Liu and colleagues in 2023. Mechanistically, this is closely related to the Transformer's attention distribution and positional encoding — during pretraining, models encounter far fewer long-text samples than short-text ones, resulting in insufficient training for processing information in the middle of long sequences. This finding directly explains OSWorld 2.0's evaluation results: during task execution exceeding 300 steps, user constraints recorded at step 50 are very likely to be "forgotten" by step 200 — not because the context window is too small, but because the attention mechanism cannot effectively retrieve key information at long distances.

Working Memory theory from cognitive science provides a useful lens for understanding this problem. Cognitive scientist Alan Baddeley's working memory model divides the human short-term information processing system into three components — the phonological loop, visuospatial sketchpad, and central executive — whose core characteristic is limited capacity that can be extended through external tools. Humans overcome working memory capacity limitations in complex tasks through external tools like sticky notes, checklists, and calendars. This directly inspires the future direction of AI Agent architecture: the need for similar "external working memory" modules that store task state in structured form outside the model's context, with on-demand retrieval and updating. During task execution exceeding 300 steps, constraints raised early by users, content from emails that arrive mid-task, and the states of previously completed steps all need to be continuously tracked and referenced throughout execution. This goes beyond a simple "memory capacity" issue — it requires a structured working memory mechanism.

Typical model mistakes include: omitting constraints explicitly stated by users; ignoring new information that appears during task execution; guessing directly when uncertain rather than proactively asking; and skipping necessary verification steps. More noteworthy is that throughout the entire task execution process, fewer than 7% of steps are used for detecting and repairing the model's own errors.

These weaknesses are concentrated in scenarios requiring recovery and maintenance of hidden states, such as implicit state inference, multi-item state tracking, conflict resolution, and dynamic environment adaptation. Looking at the distribution of task tags, the most common challenges are cross-source reasoning (42.6%), visual-spatial precision (41.7%), and implicit state inference and multi-item state tracking each at 39.8% — precisely the areas where current AI is weakest. Solutions currently being explored in academia include structured state databases (storing task constraints, completed steps, and items pending verification in categorized form), active summarization mechanisms (periodically compressing and extracting key information from context), and memory-augmented architectures (such as MemGPT, which brings operating system virtual memory management concepts into LLM context management). However, no mature solution has yet demonstrated stable performance in real long-horizon scenarios. The OSWorld 2.0 finding that only 7% of steps are used for self-correction provides a clear quantitative benchmark for this research direction.

Direction Matters More Than Parameters

The most essential value of OSWorld 2.0 lies in advancing AI computer operation evaluation from short tasks into real long-horizon work scenarios, bursting the bubble of high accuracy on short tasks and enabling the industry to see the true capability boundaries of current AI Agents.

It also clearly points to future R&D directions: the path forward is not simply piling on model parameters or increasing step budgets, but improving AI's long-horizon state maintenance, error self-detection and repair, dynamic environment adaptation, and proactive interaction capabilities. Task economic value mapping shows that these workflows cover real high-value domains including document preparation (23.8%), software and databases (17.8%), and financial operations and analysis (16.3%). Only once these are conquered can AI truly take on complex work spanning half a day.

OSWorld 2.0 is now fully open-sourced, including the environment, 108 tasks, self-hosted websites, and AI execution traces, providing a solid foundation for future research. We still have a long way to go before we reach AI agents that can truly "take over half a day's work."