Automation-Bench
An evaluation benchmark focused on browser and desktop automation capabilities.
Timeline (last 90 days)
On the four benchmarks ProgramBench, NL2Repo-Bench, Automation-Bench and Agents' Last Exam, V4.1 Flash 1017 scores 29.1, 68.2, 60.5 and 35.4 respectively, all higher than V4 Pro 0813's 15.5, 61.5, 43.2 and 25.7.
According to official self-reported data, compared with V4.1-Flash, V4.1-Flash-1017 improved on Terminal-Bench 4.0 from 31.2 to 42.3 (+11.1), on HLE from 36.8 to 42.6 (+5.8), on DeepSWE v1.1 from 74.2 to 77.0 (+2.8), and on Automation-Bench from 54.8 to 60.5 (+5.7).
All Facts (2)
On the four benchmarks ProgramBench, NL2Repo-Bench, Automation-Bench and Agents' Last Exam, V4.1 Flash 1017 scores 29.1, 68.2, 60.5 and 35.4 respectively, all higher than V4 Pro 0813's 15.5, 61.5, 43.2 and 25.7.
50%UnverifiedAccording to official self-reported data, compared with V4.1-Flash, V4.1-Flash-1017 improved on Terminal-Bench 4.0 from 31.2 to 42.3 (+11.1), on HLE from 36.8 to 42.6 (+5.8), on DeepSWE v1.1 from 74.2 to 77.0 (+2.8), and on Automation-Bench from 54.8 to 60.5 (+5.7).
50%