New LSCBench Benchmark: Coding Agents Pass 91% of Tasks, Only 25% Are Secure

How strong are coding agents at writing code? The new LSCBench benchmark delivers a set of unsettling numbers: functional pass rates reach as high as 91.3%, but at most only 25.1% of tasks both run correctly and pass security checks.

A Real Case: 14/14 Functional Tests Pass, 4/4 Security Tests Fail

Researchers from the National University of Singapore and Alibaba Group (including the Qwen team) submitted the Last Secure Code Benchmark (LSCBench) to arXiv. A Java config importer case in the benchmark is highly representative: the agent rewrote the missing core file, and all 14 hidden functional tests passed, but all 4 security tests failed — because it used SnakeYAML's default constructor, which "executes every !!class tag in the uploaded YAML."

With the same input, Claude Fable 5 and GPT-5.6-Sol parsed using SafeConstructor and passed both sets of tests. Kimi-K3, Qwen3.8-Max, and GLM-5.3 all made the same mistake. This case shows that all-green functional tests do not mean the code is secure.

SnakeYAML and Unsafe Deserialization

SnakeYAML is a widely used YAML parsing library in the Java ecosystem. Its default load() or new Yaml() construction supports YAML's !! type tags, allowing arbitrary Java classes to be specified in the document and triggering their instantiation. As long as an attacker can write gadget chains like !!javax.script.ScriptEngineManager or !!java.net.URLClassLoader into the YAML content, they may be able to load remote bytecode into the target process and execute arbitrary commands. This class of vulnerability corresponds to CWE-502 (Deserialization of Untrusted Data). The defense is to explicitly use SafeConstructor, a Constructor whitelist, or LoaderOptions to restrict which types can be parsed. This case is typical because functional tests only care whether the YAML can be correctly read and converted into a config object; they do not check whether the parser restricts type tags. Therefore, code using the default constructor can easily pass all functional tests while failing every security test.

What LSCBench Actually Tests

The benchmark contains 450 tasks derived from 150 real vulnerabilities, covering seven languages and 24 CWE categories. Of these, 115 scenarios come from public CVEs, and 35 are Java applications written by security experts.

The key lies in the prompt design: each task only requires implementing functionality, and the specification never mentions security. This is exactly how most developers use coding agents — you need an import feature, not an import feature that defends against deserialization attacks.

Each task has three scales: filling in masked functions, writing deleted files, or rebuilding a cleared source tree from the specification. A task only scores if both hidden test suites — functional and security — pass. There are no partial scores and no judge model.

CWE and Real-Vulnerability Benchmarks

CWE (Common Weakness Enumeration) is a software weakness classification system maintained by MITRE that assigns an identifier to each type of flaw, such as CWE-79 cross-site scripting, CWE-89 SQL injection, and CWE-502 unsafe deserialization. LSCBench covers 24 CWE categories, meaning it does not test just one class of vulnerability but spans multiple defect families including injection, memory safety, access control, and information disclosure. Of the benchmark scenarios, 115 come from public CVEs (Common Vulnerabilities and Exposures), i.e., real vulnerabilities that have been assigned identifiers and publicly disclosed; the other 35 are Java applications specifically written by security experts. Using real vulnerabilities rather than artificially simplified problems has the advantage of keeping tasks close to engineering reality, at the cost of relying on hidden tests being correct and raising the barrier to reproduction.

Leaderboard: Kimi-K3 Is Strongest Functionally, Weakest on Security

All six frontier models ran at maximum reasoning effort, five in Claude Code v2.1.143 and GPT-5.6-Sol in Codex v0.154.0. The results are as follows:

Model	Functional pass rate	Functional + security pass rate
Claude Fable 5	91.1%	25.1%
GPT-5.6-Sol	89.1%	24.7%
Qwen3.8-Max	89.3%	22.0%
GLM-5.3	87.1%	21.3%
DeepSeekAd-V4-Pro	73.6%	18.0%
Kimi-K3	91.3%	16.4%

The most noteworthy row is Kimi-K3: highest functional pass rate, lowest security pass rate. A green test suite tells you nothing about which agent is safer. Across all 2,700 runs, 66% of results were code that works but still has vulnerabilities. Of the 150 scenarios, 57 could not be completed securely by any model.

Where the Code Goes Wrong

By language, the combined functional + security pass rate across the six models: C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10%.

By vulnerability family, the gap is even larger: memory safety is highest at 44%; injection classes 28%; server-side request forgery (SSRF) only 3%, and information exposure only 1%. The paper's case studies show specific scenarios:

• Command injection (Java): Four of the six models passed the mount argument to ProcessBuilder as a list, while Qwen3.8-Max and Kimi-K3 built a string for sh -c.
• SSRF (Go): In a LocalAI task, GPT-5.6-Sol's GetURI would read any path passed in, including file:// URLs. The reference fix restricts reads to the model directory, but no model did this.
• Information exposure (Go): In a syft task, GLM-5.3 followed the tag conventions of an adjacent struct, causing authentication passwords to appear in config dumps and SBOM output. All six models made the same choice.

These problems are not obscure, and an experienced reviewer could absolutely spot them.

Why SSRF and Information Exposure Are the Hardest

SSRF (server-side request forgery) refers to a program accepting an externally controllable address parameter and then making requests on the attacker's behalf to internal or unintended locations. Typical paths include file:// to read local files, http://169.254.169.254 to obtain cloud metadata, or probing services through intranet addresses. Its defense cannot rely on input filtering alone; it usually requires whitelists, post-resolution validation of host and protocol, and redirect restrictions. Information exposure is more insidious: sensitive fields such as passwords and tokens are not actively leaked, but are accidentally serialized into config dumps, logs, or SBOM output. In LSCBench, these two categories have functional + security pass rates of only 3% and 1%, respectively, because agents often implement the data path only according to the functional spec and lack the context of "which fields should not appear in output," while the hidden tests precisely check whether passwords appear in exported results.

How Much Does One Reminder Help

The researchers appended one sentence to four models requiring them to implement functionality "while ensuring security." The functional + security pass rates rose significantly: GPT-5.6-Sol from 24.7% to 46.2%, GLM-5.3 from 21.3% to 38.2%, Qwen3.8-Max from 22.0% to 37.6%, and DeepSeek-V4-Pro from 18.0% to 19.1%.

Functional pass rates changed little. The paper's conclusion is: "Even with the reminder, most tasks remain insecure," with SSRF and information exposure still at the bottom. The three models improved by roughly 16 to 22 percentage points, but there is still a long way to go before truly secure coding.

Practical Strategies You Can Use

Add a security reminder to your AGENTS.md or CLAUDE.md. The paper appended it after the task, but the closest everyday practice is to write it into project configuration. For three of the four models, this yields an improvement of about 16 to 22 percentage points.

Then write tests the way the benchmark does: replay attacks. If the agent builds a loader that accepts a model name, check whether it rejects URLs and directory traversal paths. For example:

from pathlib import Path
import pytest

MODELS = Path("models").resolve()

def resolve_model(uri: str) -> Path:
    if "://" in uri:
        raise ValueError("plain model names only")
    p = (MODELS / uri).resolve()
    if not p.is_relative_to(MODELS):
        raise ValueError("outside the models dir")
    return p

@pytest.mark.parametrize("uri", [
    "ftp://example.com/model.bin",
    "../secrets.txt",
])
def test_rejects_escape(uri):
    with pytest.raises(ValueError):
        resolve_model(uri)

When reviewing an agent's work, deliberately review security in small chunks. Splitting a 40-file change by round or by file and reading it chunk by chunk makes SSRF-type issues easier to spot than scanning the whole thing at once.

You can also run this benchmark on your own agent configuration. It is released as a Harbor dataset (LSCBench itself is Apache 2.0) and requires Harbor 0.24+, Docker, Python 3.12+, and about 58 GB of task images. The project contains exploitable vulnerabilities, and the README recommends running it in an isolated environment.

Embedding Security Constraints in AGENTS.md

AGENTS.md and CLAUDE.md are instruction files that coding agents read from the project root, and their contents are injected as system-level context into every conversation turn. Writing security requirements here is more stable than appending a reminder at the end of a single task: it takes effect when the agent begins planning, rather than waiting until the code is written and then having the model go back and patch it. Effective writing should be as specific as possible, for example: "All YAML parsing must use SafeConstructor," "Reject inputs containing :// or .. when loading model names," and "Remove password, token, and secret fields before exporting configuration." Avoid unenforceable abstract requirements like "please ensure security." For the four models in this experiment, such prompts yielded about 16 to 22 percentage points of improvement in functional + security pass rates, but they are still insufficient to cover the two hardest vulnerability classes, SSRF and information exposure.

Details to Note Before Citing the Data

The leaderboard tests model versions available at the time the paper was written, and newer versions may score differently. All numbers are pass@1, with only one attempt per task. Qwen3.8-Max is both one of the six models being scored and a builder agent that wrote tests and specifications; the authors believe its mid-range performance indicates no advantage. The paper is currently an arXiv preprint and should be treated cautiously until someone reproduces it.