Simular
A company focused on computer-use AI agents that publicly releases run data and reproduction results for benchmarks such as OSWorld.
Timeline (last 90 days)
On the same benchmark, taking the best run versus taking the average can differ by 2.6 percentage points; Simular's data shows 0.7674 by the first clean run and 0.7938 by the best run.
While generating the OSWorld 2.1 reference runs, Simular had tasks stop after one step due to the model key's usage limit, yet they were recorded as successful with a score of 0, and the average score over the whole folder was computed as 0.40.
Simular ran the same agent on two desktop images, one scoring 0.7766 and the other 0.7414.
All Facts (3)
On the same benchmark, taking the best run versus taking the average can differ by 2.6 percentage points; Simular's data shows 0.7674 by the first clean run and 0.7938 by the best run.
50%UnverifiedSimular ran the same agent on two desktop images, one scoring 0.7766 and the other 0.7414.
50%UnverifiedWhile generating the OSWorld 2.1 reference runs, Simular had tasks stop after one step due to the model key's usage limit, yet they were recorded as successful with a score of 0, and the average score over the whole folder was computed as 0.40.
50%