Program Bench
A benchmark for evaluating the programming capabilities of AI models, used for side-by-side comparison of code generation and programming task performance.
Timeline (last 90 days)
DeepSeek V4.1 Flash 1017 rose from 31.2 to 42.3 on Terminal Bench 4.0, from 20.2 to 42.3 on Program Bench, from 31.8 to 35.4 on Agent's Last Exam, from 64.6 to 72.1 on Apex Bench, from 78.9 to 83.0 on Chartography, and from 89.6 to 92.5 on Baby Vision.
The Program Bench score rose from 20.3 to 29.1, an improvement of 8.8 points, using the 'Almost' criterion (passing 5 of 9 tests counts as a pass).
When a third party tested the September version using the strictest 'Fully' criterion, the Program Bench pass rate was only 0.5%.
All Facts (3)
DeepSeek V4.1 Flash 1017 rose from 31.2 to 42.3 on Terminal Bench 4.0, from 20.2 to 42.3 on Program Bench, from 31.8 to 35.4 on Agent's Last Exam, from 64.6 to 72.1 on Apex Bench, from 78.9 to 83.0 on Chartography, and from 89.6 to 92.5 on Baby Vision.
50%UnverifiedThe Program Bench score rose from 20.3 to 29.1, an improvement of 8.8 points, using the 'Almost' criterion (passing 5 of 9 tests counts as a pass).
50%UnverifiedWhen a third party tested the September version using the strictest 'Fully' criterion, the Program Bench pass rate was only 0.5%.
50%