OS World
OSWorld is a benchmark framework for evaluating the computer operation capabilities of AI agents, focused on measuring a model's ability to complete tasks within real desktop operating system environments, covering interaction scenarios such as file management, application operation, and web browsing. The benchmark simulates real user operation workflows to assess a model's planning, execution, and autonomous decision-making capabilities, serving as an important reference for measuring the overall performance of computer-use agents.
Core Facts
OpenAI officially claims that GPT-4.1 Sol is smarter than Astra in certain respects.
Qwen3 27B's score on OS World Verified jumped from 63.9 to 84.3 (self-reported official data).
On the DeepSuite benchmark, Grok 4.7 scored 71%, GPT 5.6 Sol scored 72.7%, Fable 5.1 scored 70%, and Astra Max topped the leaderboard at 74.1%.
Timeline (last 90 days)
On the OS World benchmark, Muse Wonderful scored 78.4, Muse Wonder scored 72.6, Fable 5.1 scored 80, Astra scored 75, and Opus scored 81.
Haiku 5.5 outperforms GPT-6 Luna by approximately 40% on Computer Use tasks.
Haiku 5.5's score on computer use tasks jumped from 16% to 72%
On the OS World benchmark, less than 12 months ago Claude scored below 50%, whereas Opus 4.7 has reportedly reached 78%.
OpenAI officially claims that GPT-4.1 Sol is smarter than Astra in certain respects.
OS World expanded the CUA battlefield from web pages to full desktop operations, supporting multiple applications such as Linux and Windows, and became one of the most standard CUA benchmarks.
On the DeepSuite benchmark, Grok 4.7 scored 71%, GPT 5.6 Sol scored 72.7%, Fable 5.1 scored 70%, and Astra Max topped the leaderboard at 74.1%.
In the OS World computer operation test, Luna scored 53%, while Astra reached 73%.
The model's official benchmark scores are SWE Bench Pro 61.7, LiveCode Bench V6 90.3, and OS World 84.3.
NextN 2.5 achieved an OS World score of 56.4 (per official claims), but the weights are marked as Coming soon and only a free hosted endpoint is provided, making independent reproduction impossible.
6 more timeline events
All Facts (20)
OpenAI officially claims that GPT-4.1 Sol is smarter than Astra in certain respects.
70%VerifiedQwen3 27B's score on OS World Verified jumped from 63.9 to 84.3 (self-reported official data).
70%VerifiedOn the DeepSuite benchmark, Grok 4.7 scored 71%, GPT 5.6 Sol scored 72.7%, Fable 5.1 scored 70%, and Astra Max topped the leaderboard at 74.1%.
65%VerifiedOS World is a benchmark introduced by Carnegie Mellon University and other institutions that tests the ability of AI agents to perform complex tasks in real operating system environments.
65%UnverifiedClaude第一次推出计算机使用能力时，OS World得分还不到30%
80%UnverifiedOS World基准测试由学术界于2024年提出，专门测试AI模型在真实操作系统环境中完成复杂任务的能力
80%Unverified人类在OS World测试中的平均得分约为72-74%
70%UnverifiedThe previous-generation Sonnet 4.5 scored 61.4% on the OSWorld Verified benchmark.
60%UnverifiedOn the OS World benchmark, Muse Wonderful scored 78.4, Muse Wonder scored 72.6, Fable 5.1 scored 80, Astra scored 75, and Opus scored 81.
50%UnverifiedHaiku 5.5 outperforms GPT-6 Luna by approximately 40% on Computer Use tasks.
50%UnverifiedHaiku 5.5's score on computer use tasks jumped from 16% to 72%
50%UnverifiedOn the OS World benchmark, less than 12 months ago Claude scored below 50%, whereas Opus 4.7 has reportedly reached 78%.
50%UnverifiedOS World expanded the CUA battlefield from web pages to full desktop operations, supporting multiple applications such as Linux and Windows, and became one of the most standard CUA benchmarks.
50%UnverifiedIn the OS World computer operation test, Luna scored 53%, while Astra reached 73%.
50%UnverifiedThe model's official benchmark scores are SWE Bench Pro 61.7, LiveCode Bench V6 90.3, and OS World 84.3.
50%UnverifiedNextN 2.5 achieved an OS World score of 56.4 (per official claims), but the weights are marked as Coming soon and only a free hosted endpoint is provided, making independent reproduction impossible.
50%UnverifiedWhen showcasing its OS World leaderboard results, Meta used test sets from different time periods to compare against older models, artificially inflating the magnitude of improvement.
50%UnverifiedOn Terminal-Bench 4.0, the standard version of Claude 5.1 scored 55.8% and the trusted version reached 60.9%; the OS World 2.0 partial completion rate was 77.9%.
50%UnverifiedCompared to the previous generation Qwen3.6-27B, Qwen3.8-27B shows significant improvements on multiple benchmarks, with Deep SWE rising from 13.3 to 42.2 (nearly tripling) and OS World rising from 63.9 to 84.3.
50%UnverifiedOS World, developed by Carnegie Mellon University and other institutions, requires models to complete real operational tasks within actual operating system virtual machines.
50%