Benchmark
NL2Repo-Bench
A benchmark that measures the ability to generate a runnable code repository from a natural language specification.
Timeline (last 90 days)
Oct 19
On the four benchmarks ProgramBench, NL2Repo-Bench, Automation-Bench and Agents' Last Exam, V4.1 Flash 1017 scores 29.1, 68.2, 60.5 and 35.4 respectively, all higher than V4 Pro 0813's 15.5, 61.5, 43.2 and 25.7.
Unverified50%