LSCBench
A code benchmark for evaluating the security of programming agents, comprising 450 tasks derived from real vulnerabilities and covering multiple languages and 24 CWE categories. A task scores only when both functional and security tests pass.
Timeline (last 90 days)
Researchers from the National University of Singapore and Alibaba Group (including the Qwen team) submitted the Last Secure Code Benchmark (LSCBench) to arXiv.
The LSCBench benchmark contains 450 tasks, derived from 150 real vulnerabilities, covering seven languages and 24 CWE categories.
Of the 150 scenarios in LSCBench, 115 come from public CVEs and 35 are Java applications written by security experts.
LSCBench evaluated six frontier models at the highest reasoning effort, with five running in Claude Code v2.1.143 and GPT-5.6-Sol running in Codex v0.154.0.
In LSCBench, the six models' functional pass rate reached as high as 91.3%, but the highest simultaneous functional and security pass rate was only 25.1%.
In LSCBench, Kimi-K3 had the highest functional pass rate (91.3%), but the lowest functional + security pass rate (16.4%).
Of all 2,700 LSCBench runs, 66% of the results were code that works functionally but still contains vulnerabilities.
Of LSCBench's 150 scenarios, 57 could not be completed securely by any model.
In LSCBench, by vulnerability family, the highest combined functional + security pass rate for memory safety was 44%, for injection classes 28%, for SSRF only 3%, and for information exposure only 1%.
In LSCBench's Java config importer case, the agent rewrote the missing core file, and all 14 hidden functional tests passed, but all 4 security tests failed, because it used SnakeYAML's default constructor, which executes every !!class tag in the uploaded YAML.
3 more timeline events
All Facts (13)
In LSCBench's Go language syft task, GLM-5.3 followed the tag convention of an adjacent struct, causing the authentication password to appear in the config dump and SBOM output, and all six models made the same choice.
50%UnverifiedThe LSCBench benchmark contains 450 tasks, derived from 150 real vulnerabilities, covering seven languages and 24 CWE categories.
50%UnverifiedOf the 150 scenarios in LSCBench, 115 come from public CVEs and 35 are Java applications written by security experts.
50%UnverifiedLSCBench evaluated six frontier models at the highest reasoning effort, with five running in Claude Code v2.1.143 and GPT-5.6-Sol running in Codex v0.154.0.
50%UnverifiedIn LSCBench, the six models' functional pass rate reached as high as 91.3%, but the highest simultaneous functional and security pass rate was only 25.1%.
50%UnverifiedIn LSCBench, Kimi-K3 had the highest functional pass rate (91.3%), but the lowest functional + security pass rate (16.4%).
50%UnverifiedOf all 2,700 LSCBench runs, 66% of the results were code that works functionally but still contains vulnerabilities.
50%UnverifiedOf LSCBench's 150 scenarios, 57 could not be completed securely by any model.
50%UnverifiedIn LSCBench, by vulnerability family, the highest combined functional + security pass rate for memory safety was 44%, for injection classes 28%, for SSRF only 3%, and for information exposure only 1%.
50%UnverifiedIn LSCBench's Java config importer case, the agent rewrote the missing core file, and all 14 hidden functional tests passed, but all 4 security tests failed, because it used SnakeYAML's default constructor, which executes every !!class tag in the uploaded YAML.
50%UnverifiedLSCBench's conclusion is that even with security reminders, most tasks remain insecure, with SSRF and information exposure still at the bottom.
50%UnverifiedIn LSCBench's Go language LocalAI SSRF task, GPT-5.6-Sol's GetURI reads any path passed in, including file:// URLs; the reference fix restricts reads to within the model directory, but no model did this.
50%UnverifiedResearchers from the National University of Singapore and Alibaba Group (including the Qwen team) submitted the Last Secure Code Benchmark (LSCBench) to arXiv.
50%