Unverified50% confidenceFactExact time
OSWorld maintainers have an unmerged fix targeting 14 of the 108 tasks, whose scorers turn a failed call to the judge model into a score of 0.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/12/2026
First Seen
Valid until: 1/10/2027
Sources
Related Entities
Related Claims
UnverifiedWhile generating the OSWorld 2.1 reference runs, Simular had tasks stop after one step due to the model key's usage limit, yet they were recorded as successful with a score of 0, and the average score over the whole folder was computed as 0.40.73% similarUnverifiedOSWorld 2.0中模型评估仅占总得分的11.53%，且没有任何任务的模型评估占比超过50%67% similarUnverifiedThe 82 tasks in OS World were not run under identical conditions—half of the tasks retained the original baseline scores, and the invested resources and number of attempts also differed—so the 7-percentage-point improvement cannot be entirely attributed to memory.65% similarUnverifiedAmong the OSWorld-Verified results tracked by Yutori, of the 111 records annotated with the number of runs, 94 were run only once.64% similarUnverifiedIn the τ-bench retail environment benchmark test (30 test tasks, simulated customers, single run, solvi 0.8.0, gpt-oss-120b as the Agent), without the guard 18 tasks were solved and the environment rejected 10 mutating calls; with the guard 14 tasks were solved and the environment rejected 0 mutating calls.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1009489API
curl https://kongchang.com/api/v1/knowledge/claims/1009489MCP
get_claim(id=1009489)