Unverified50% confidenceFactExact time
Among the OSWorld-Verified results tracked by Yutori, of the 111 records annotated with the number of runs, 94 were run only once.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/12/2026
First Seen
Valid until: 1/10/2027
Sources
Related Entities
Related Claims
UnverifiedOSWorld maintainers have an unmerged fix targeting 14 of the 108 tasks, whose scorers turn a failed call to the judge model into a score of 0.64% similarUnverifiedAcross six runs, 970 out of 972 gold event instances received overlapping predictions, indicating that models can almost always locate event positions.59% similarUnverifiedThe test suite consists of 10 benchmarks; most run 1,000 questions each, with HumanEval comprising 164 questions and TruthfulQA comprising 817 questions.57% similarUnverifiedThere were 100 tasks on which performance losses relative to the baseline persisted across all three repetitions, while persistent wins occurred on only 1 task.57% similarUnverifiedThe 82 tasks in OS World were not run under identical conditions—half of the tasks retained the original baseline scores, and the invested resources and number of attempts also differed—so the 7-percentage-point improvement cannot be entirely attributed to memory.56% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1009480API
curl https://kongchang.com/api/v1/knowledge/claims/1009480MCP
get_claim(id=1009480)