AI Coding Agents Pass 91% of Tests, But Only 25% Are Safe to Ship

When your AI coding assistant confidently hands over code that passes every test, it may have just planted a time bomb. The LSCBench benchmark, jointly submitted by the National University of Singapore and Alibaba (including the Qwen team), uses 150 real vulnerabilities to expose this harsh truth.

Two Users, Two Sets of Exams

Every feature has two kinds of audiences. Friendly users want the feature to work correctly—upload a config, complete an import. Functional tests simulate exactly this kind of user. Hostile users want the feature to break in some useful way—say, uploading a config that quietly loads a class it shouldn't. Security tests play this role.

The same piece of code, two examiners. Coding agents are trained and prompted to serve friendly users, because that's who the spec describes. LSCBench is the first large-scale benchmark to systematically ask, "What happens when the second examiner shows up too?"

Scoring: Reviewing Code Like an Attacker

The benchmark is deliberately strict, otherwise agents could easily game it:

• 450 tasks drawn from 150 real vulnerabilities: 115 from public CVEs, 35 from Java applications written by security experts, covering 7 languages and 24 weakness categories (CWEs).
• Three task scales: fill in a masked function, write a deleted file, or rebuild a cleared source tree from a spec.
• Prompts never mention security. In the authors' words: every task asks only for functionality, and security is never mentioned in the spec. This is exactly how most people actually use coding agents—you want the import feature, not a "deserialization-safe" import feature.
• Dual hidden test suites. Each task only scores when both the functional test and the security test pass, and the security test replays the original attack. No partial credit, no LLM judge, one attempt per task (pass@1).

Six frontier agents run at maximum reasoning effort, five of them inside Claude Code v2.1.143, and GPT-5.6-Sol inside Codex v0.154.0.

CWE (Common Weakness Enumeration) is a software weakness classification system maintained by MITRE. Unlike CVE, which targets specific products, CWE describes reusable defect patterns of the "why does code go wrong" variety—for instance, CWE-78 corresponds to OS command injection, CWE-918 to SSRF, and CWE-200 to information exposure. LSCBench's coverage of 24 CWE categories means its 150 tasks are not a random sample, but deliberately span weakness families from memory safety and injection to SSRF and information exposure, so it can answer "which weakness category do agents systematically fail on." Reading the leaderboard with CWE as the axis lets you avoid structural differences hidden by a single aggregate score—this is exactly how the cliff of SSRF at just 3% and information exposure at just 1% gets exposed in this article.

pass@1 is the standard scoring convention in code generation evaluation: the model generates only one answer per task, scores 1 if it passes and 0 if it fails, and the final score is the average across all tasks. Its key difference from pass@k (which allows k generations and counts success if any one passes) is that pass@k measures the "upper bound of model capability," while pass@1 measures the "success rate of a single delivery." In real development you usually only get one output from an agent, so pass@1 is closer to production reality. LSCBench's insistence on pass@1, no partial credit, and no LLM judge is precisely to close off the two shortcuts of "try a few times and get lucky" and "fuzzy scoring that lets things slide," so the 91-to-25 gap cannot be diluted by the evaluation design itself.

Dual hidden test suites means each task comes with two sets of tests not disclosed to the model: one functional test that reproduces the normal usage path, and one security test that replays the original exploit. Hidden means the agent cannot reverse-engineer the test points by reading test code—it can only rely on its own prior knowledge to guess where things might be unsafe. This design turns evaluation from "write code to match the tests" into "write code without knowing how the examiner will attack," and it is the engineering implementation of the "second examiner" metaphor repeatedly emphasized in this article.

Leaderboard: The 91 vs. 25 Chasm

Model	Tool	Functional Pass Rate	Functional + Security Pass Rate
Claude Fable 5	Claude Code	91.1%	25.1%
GPT-5.6-Sol	Codex	89.1%	24.7%
Qwen3.8-Max	Claude Code	89.3%	22.0%
GLM-5.3	Claude Code	87.1%	21.3%
DeepSeekAd-V4-Pro	Claude Code	73.6%	18.0%
Kimi-K3	Claude Code	91.3%	16.4%

The last row deserves repeated chewing. Kimi-K3 has both the highest functional pass rate and the lowest security pass rate. A string of green test results simply cannot tell you which agent is safer.

The aggregate data is more shocking than any single row: across all 2,700 runs, 66% produced code that runs but still contains vulnerabilities; 57 of the 150 scenarios were never safely solved by any model—not a single one of the six frontier agents escaped.

Where the Code Breaks: Languages and Weakness Families

By language (share of all runs passing both functional and security, across six models): C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10%.

By weakness family (under the stricter criterion of using code that passes functionally as the denominator): memory safety performs best at 44%; injection 28%; then a cliff—SSRF just 3%, information exposure just 1%.

That cliff is the key. Memory safety fixes are mostly local—add a bounds check, free a pointer. But SSRF and information exposure require the agent to model the system: where can this URL reach? Who else reads this config dump? The agent finished the feature but left the door wide open.

Case studies in the paper show what this looks like in practice:

• Command injection, Java: four of six models passed the mount argument to ProcessBuilder as a list. Qwen3.8-Max and Kimi-K3 instead built a string for sh -c—the shell interprets whatever the user inputs as commands.
• SSRF, Go (LocalAI task): GPT-5.6-Sol's GetURI reads any path handed to it, including file:// URLs. The reference fix restricts reads to the model directory, but no model did this.
• Information exposure, Go (syft task): GLM-5.3 followed the tag convention of the adjacent struct, and an auth password ended up in the config dump and SBOM output. All six models made the same choice.

None of these are esoteric issues. A reviewer deliberately looking for them would catch all three. The agents weren't looking.

SSRF (Server-Side Request Forgery) refers to a program accepting a user-controllable URL or path and using the server's own privileges to make requests or read resources. The danger is not only accessing external hosts, but also that protocols and addresses like file://, gopher://, and http://169.254.169.254/ can let an attacker read local files, access cloud metadata services, or probe the internal network. The core of defense is not a simple blacklist, but whitelist-style normalized validation of protocol, host, port, and path, and restricting file reads to preset directories. In this article, GPT-5.6-Sol's GetURI reads any path given to it (including file://), while the correct fix only allows reads within the model directory—this explains why SSRF's security pass rate is as low as 3%: it requires the agent to understand "where this URL can ultimately reach," a system-level inference rather than a one-line bounds check.

Information exposure (CWE-200) refers to sensitive data being read, logged, or output in unexpected places, with typical scenarios being auth passwords, tokens, and internal paths appearing in config dumps, logs, or SBOM output. It is hard to prevent because the problem is not in some function, but in the data propagation path: once a field is given a serialization tag, it may be automatically collected by multiple exporters, and each exporter individually "did nothing wrong." In this article, GLM-5.3 merely followed the tag convention of an adjacent struct, and an auth password flowed into the config dump and SBOM output, and all six models did the same. This kind of weakness requires the agent to proactively ask "who else will read this data," and that is precisely the global perspective current agents lack most.

A One-Line Prompt Helped, But Didn't Solve the Problem

The authors reran tests on four of the six models, appending one sentence after each task—implement to meet the functional requirement, while "also being safe":

• GPT-5.6-Sol: 24.7% → 46.2%
• GLM-5.3: 21.3% → 38.2%
• Qwen3.8-Max: 22.0% → 37.6%
• DeepSeek-V4-Pro: 18.0% → 19.1%

Functional pass rates fluctuated by a few percentage points. The paper's conclusion: even with the reminder, most tasks remain unsafe—SSRF and information exposure stay near the bottom. (Claude Fable 5 and Kimi-K3 did not participate in the rerun.)

Five Takeaways

1. Green tests ≠ shippable. Engrave the 91/25 gap in your mind: a passing test suite measures friendly users. If your agent's output reaches places hostile users can touch, you need a second test suite.
2. The failure mode is omission, not error. Safe APIs are always available—SafeConstructor, argv lists, path restrictions. Agents just never reach for them. To an agent, security is a retrieval problem: the knowledge is in the weights, but the trigger is missing.
3. Write attack tests, not just functional tests. LSCBench's entire trick is replaying the original exploit. For your own agent-generated code: if it accepts a model name, check whether it rejects URLs and ../ path traversal.
4. Add that reminder regardless. One sentence nearly doubled safe output for three of four models (+16 to +22 percentage points). Adding a line to your AGENTS.md or CLAUDE.md is the closest everyday equivalent—low cost, no downside in the data, just don't mistake it for solving the problem.
5. Read the fine print. Everything is pass@1, one attempt. Qwen3.8-Max was also the agent that built the tests and specs—the authors believe its mid-tier performance shows it gained no advantage. This is an arXiv preprint: treat it as one data point until someone reproduces it.

Source: LSCBench paper (arXiv, NUS + Alibaba); LSCBench project page. Data cited from the authors' public leaderboard. dev.to author Gabriel Anhaia's write-up first disclosed several of these details.

One-line Summary

LSCBench shows six frontier coding agents pass functional tests at ~91%, but only 25% pass security.

Paragraph Summary

The LSCBench benchmark, jointly released by the National University of Singapore and Alibaba, uses 150 vulnerability tasks derived from real CVEs and security expert-written code to systematically test the security capabilities of six frontier coding agents. The results are striking: functional test pass rate reaches 91%, but the rate of passing both functional and security tests is only 25%; across 2,700 runs, 66% of code runs but contains vulnerabilities, and 57 of 150 scenarios were never safely solved by any model. Most strikingly, Kimi-K3 has the highest functional pass rate (91.3%) and the lowest security pass rate (16.4%), showing green test results cannot reflect real security. The failure mode is mostly omission rather than error—safe APIs like SafeConstructor, argv lists, and path restrictions are always available, but agents never proactively call them. SSRF and information exposure are nearly impossible to defend against, because adding "also be safe" to the prompt can nearly double some models' safety rates, but it still falls far short of solving the problem.