OSWorld
An operating system-level benchmark environment for AI agents, designed to evaluate agent performance on desktop operating system tasks. It provides a standardized setting for assessing how well autonomous agents can interact with and complete tasks within general-purpose desktop computing environments.
Core Facts
Timeline (last 90 days)
OSWorld maintainers have an unmerged fix targeting 14 of the 108 tasks, whose scorers turn a failed call to the judge model into a score of 0.
Between OSWorld 2.0 and 2.1, 174 pull requests targeting task files were merged.
After fixing the bug in the zoom tool and raising the per-round token limit, Anthropic re-ran the old model, and Claude Opus 4.7's OSWorld score rose from 78.0% to 82.8%, with no change to the model itself.
An OSWorld PR shows that 357 of 23,244 actions were executed twice because the controller retried commands that had already been partially executed.
After the OSWorld team compressed the image from 50 GB to 25 GB, LibreOffice's first launch dropped from one minute to 5 seconds.
The new model achieved 1,620 points on GDPVal (compared to 735 for the previous generation), improved on OSWorld from 15.7% to 72.4%, on Terminal Bench from 0.0% to 39.2%, and on Humanity's Last Exam from 10.2% to 45.9% without tools and from 18.7% to 57.4% with tools.
Across multiple benchmarks, the gap between Sonnet 5.5 and Opus 5.5 is minimal, such as GDP Val at 1844 vs. 1846, OS World at 80.1% vs. 81%, and Humanity's Last Exam at 64.5 vs. 67.7.
On the DeepSuite benchmark, Grok 4.7 scored 71%, GPT 5.6 Sol scored 72.7%, Fable 5.1 scored 70%, and Astra Max topped the leaderboard at 74.1%.
Sai achieved a score of 73% on the OSWorld benchmark.
When OSWorld was released in 2024, the strongest baseline model score was only approximately 12%, while the human completion rate was approximately 72%.
5 more timeline events
All Facts (20)
On the DeepSuite benchmark, Grok 4.7 scored 71%, GPT 5.6 Sol scored 72.7%, Fable 5.1 scored 70%, and Astra Max topped the leaderboard at 74.1%.
65%UnverifiedOSWorld基准测试由卡内基梅隆大学等机构于2024年提出，包含369个涵盖Linux、Windows和macOS的真实计算机任务
90%UnverifiedOSWorld是首个面向真实操作系统环境的多模态Agent评估基准
85%Unverified在OSWorld最初发布时，最强的GPT-4V模型仅达到约12.24%的成功率
85%UnverifiedAnthropic的Computer Use在OSWorld基准测试上的操作准确率约为14.9%（2024年10月数据）
85%UnverifiedClaude achieved a task completion rate of 14.9% on the OSWorld benchmark, representing the highest level at the time.
60%UnverifiedAfter fixing the bug in the zoom tool and raising the per-round token limit, Anthropic re-ran the old model, and Claude Opus 4.7's OSWorld score rose from 78.0% to 82.8%, with no change to the model itself.
50%UnverifiedOSWorld maintainers have an unmerged fix targeting 14 of the 108 tasks, whose scorers turn a failed call to the judge model into a score of 0.
50%UnverifiedAfter the OSWorld team compressed the image from 50 GB to 25 GB, LibreOffice's first launch dropped from one minute to 5 seconds.
50%UnverifiedThe new model achieved 1,620 points on GDPVal (compared to 735 for the previous generation), improved on OSWorld from 15.7% to 72.4%, on Terminal Bench from 0.0% to 39.2%, and on Humanity's Last Exam from 10.2% to 45.9% without tools and from 18.7% to 57.4% with tools.
50%UnverifiedAcross multiple benchmarks, the gap between Sonnet 5.5 and Opus 5.5 is minimal, such as GDP Val at 1844 vs. 1846, OS World at 80.1% vs. 81%, and Humanity's Last Exam at 64.5 vs. 67.7.
50%UnverifiedOSWorld is an open evaluation benchmark jointly released by Shanghai Jiao Tong University and other institutions, comprising 369 tasks executed in real operating systems and covering software such as web browsers, file managers, and Office suites, using task success rate as the metric.
50%UnverifiedSai achieved a score of 73% on the OSWorld benchmark.
50%UnverifiedWhen OSWorld was released in 2024, the strongest baseline model score was only approximately 12%, while the human completion rate was approximately 72%.
50%UnverifiedMind2Web and OSWorld are benchmarks based on static screenshots, allowing agents to make decisions on completely static interfaces.
50%UnverifiedMainstream GUI agent benchmarks such as OSWorld and WindowsAgentArena have begun to incorporate the Accessibility Tree for state verification
50%UnverifiedQwen3 27B's performance on computer operation benchmarks is: OSWorld Verified 84.3, WebArena Verified 64.8, and AndroidWorld 81.9.
50%UnverifiedMythos Preview比Mythos 5早发布两个多月，但在UK AISI cyber ranges、GPQA、HLE+tools、OSWorld及多数病毒学任务等多项基准测试上超越了Mythos 5
50%UnverifiedAn OSWorld PR shows that 357 of 23,244 actions were executed twice because the controller retried commands that had already been partially executed.
50%UnverifiedBetween OSWorld 2.0 and 2.1, 174 pull requests targeting task files were merged.
50%