Teaching AI to Find Vulnerabilities with Reinforcement Learning: A CMU Professor Reveals the Path from Crashes to Sandbox Escapes

Cybersecurity is undergoing a profound transformation. As software ships faster than ever before, the question of how to check code security at machine speed and scale has become urgent. David Brumley — full-time professor at Carnegie Mellon University (CMU) and Chief AI & Science Officer at BugCrowd — draws on over twenty years of research to offer his answer: teach AI to think and attack like elite human hackers.

From Teaching Humans to Teaching Machines: The Essence of How Hackers Learn

Brumley's research starting point comes from a true story. In 2016, a 17-year-old using the handle "Fluorescence" suddenly appeared on the PicoCTF leaderboard. PicoCTF is a cybersecurity competition platform for middle and high school students created by CMU's CyLab Security and Privacy Institute, reaching roughly one million U.S. high schoolers. CTF (Capture The Flag) is the most popular competitive format in cybersecurity — participants solve challenges spanning reverse engineering, exploit development, cryptography, and web security within a time limit, earning a specific "flag" string upon solving each challenge. CTF competitions are crucial for security talent development because they provide a progressive learning path from simple to complex. This teenager wasn't from one of the "usual winner" schools in Palo Alto or Washington, D.C. — he came from competitive math.

When asked how he learned to hack, the teenager's answer revealed the core methodology: "I look at cybersecurity tasks, then Google the information I need, read relevant materials and writeups, then start imitating and practicing." Using this method alone, he progressed from simple problems to increasingly difficult challenges, eventually finishing second in PicoCTF.

That teenager, named Richard Zhu, was later recruited by Brumley to CMU. Just two years after he started learning cybersecurity, he became a Pwn2Own champion. Pwn2Own is a premier vulnerability exploitation contest organized by Trend Micro's Zero Day Initiative (ZDI), held annually since 2007, where participants must demonstrate zero-day attacks against fully patched commercial software live on stage. "Pwn" means breaking into a target, "Own" means gaining control — successful contestants win substantial cash prizes and keep the compromised devices. Richard Zhu was the first person to successfully hack a Tesla, earning $375,000 in cash and a brand-new Tesla with a single exploit.

Brumley's core thesis: Teaching frontier AI models to hack works exactly the same way as teaching a high schooler to become a Pwn2Own champion. This methodology can be decomposed along two dimensions: target difficulty (from toy programs and synthetic CTF challenges all the way to hardened real-world targets) and exploitation difficulty (from locating bugs and triggering crashes to achieving arbitrary read/write and full code execution).

Hacking Is Fundamentally a Ladder

"Hacking is a ladder." Brumley emphasizes that this is precisely why cybersecurity aligns so well with reinforcement learning. Reinforcement Learning (RL) is a machine learning paradigm where an agent learns optimal behavior by interacting with an environment and adjusting its strategy based on reward signals. RL is particularly well-suited to cybersecurity because the attack process naturally embodies RL's three core elements: state (the current state of the target system), action (operations the attacker can execute), and reward (whether the attack succeeds). Unlike domains such as natural language processing, attack success can typically be verified deterministically through program behavior (e.g., crashes, privilege escalation), providing high-quality reward signals for RL. We have a series of tasks with increasing difficulty, and we can usually design a reliable oracle to judge whether the model has completed a task — precisely measuring whether the model has learned the right capabilities.

RL Environments for Vulnerability Discovery: Fatal Flaws in Current Benchmarks

Designing an RL environment (or "gym") for cybersecurity requires several key components: a reproducible vulnerable program (typically containerized for consistency), a scoring oracle, and an LLM with an orchestrator.

Brumley specifically points out that "LLM as a judge" is flawed in cybersecurity — because LLMs will always claim they've successfully completed an attack. Deterministic scoring oracles must therefore be designed for each capability level. The most basic level checks "did it trigger a crash," while the highest level checks "can it hijack control flow, launch a calculator, or pop a reverse shell."

To distinguish hallucinations from real vulnerabilities, task prompts typically don't just ask "can you find a vulnerability" — instead, they require the LLM to actually exploit the vulnerability, providing a crash-triggering input as a witness.

The Collapse of the Single-Vulnerability Assumption

Existing benchmarks (like SciBench, CyberGym) harbor a fatal implicit assumption: a program contains only one vulnerability. In reality, it's nearly impossible to find programs with just a single bug.

When a program has multiple vulnerabilities, everything falls apart. If you ask "can you find and exploit a vulnerability," the LLM has enormous room for reward hacking — it will always go after the easiest bug. Reward hacking is one of RL's core challenges, where an agent finds a shortcut that satisfies the literal conditions of the reward function but violates the designer's true intent. This problem is widely discussed in AI safety alignment research: if the reward function only requires "trigger a crash," the model will repeatedly exploit the easiest-to-trigger vulnerability for reward rather than exploring deeper, more complex security flaws — similar to a game AI discovering a score exploit instead of learning actual gameplay strategy. This severely limits the model's learning trajectory and prevents it from growing smarter.

Both approaches existing benchmarks use are problematic: if you tell the LLM which function contains the bug (e.g., by providing a backtrace), it doesn't need to reason and its capability is stunted; if you don't tell it, it will always pick the easiest bug when multiple exist.

Brumley proves that "manually constructing single-vulnerability environments" doesn't work using two enormously expensive case studies: DARPA's $60 million Cyber Grand Challenge found that 50% of hand-curated challenges contained unexpected unknown vulnerabilities; and in the AIXCC project, for which he personally designed the scoring algorithm, 18 of the vulnerabilities found at DEF CON were unintended.

Audit Task: Letting the Program Define the Task

To address this challenge, Brumley's team developed a new approach called "Audit Task." The core idea is to flip the problem from "find a vulnerability" to "find all vulnerabilities."

This way, the LLM can submit multiple vulnerability proofs — both known vulnerabilities and potentially new ones you haven't discovered. All proofs are verified through a deterministic oracle, using stack backtraces to differentiate distinct vulnerabilities. A stack backtrace records the function call chain at the time of a crash — the complete sequence of function calls from program entry to crash point. Using it for vulnerability deduplication is industry standard practice: if two crashes share the same stack backtrace, they are generally considered different triggers of the same vulnerability. Microsoft's Windows Error Reporting and Apple's Crash Reporter both use similar mechanisms to cluster massive volumes of crash reports.

Scoring uses the product of precision and recall: recall measures the proportion of known vulnerabilities found, while precision prevents the model from spamming invalid false positives. This mechanism elegantly balances two competing objectives: encouraging discovery of unknown vulnerabilities while preventing junk submissions.

The advantage of this "open-world" scoring is that it can be applied directly to real open-source software with multiple vulnerabilities, can discover unknown bugs, provides a cleaner learning trajectory, and completely eliminates LLM-as-judge bias. Brumley summarizes: "Don't define the task with a single vulnerability — let the program define the task."

Validating AI Weaponization Capabilities: Can It Break Chrome V8 and Other High-Value Targets?

The second critical dimension is the model's capability ceiling. Brumley draws a clear distinction between "finding bugs" and "real hacking": Crashing a program is not the same as breaking into it — you can't steal intellectual property by making a program crash.

Yet all existing public experiments (AIXCC, CyberGym, Bounty Bench) only test whether AI can crash a program. Whether AI can actually compromise high-value targets had remained an open question.

Why Chrome V8 Was Chosen as the Test Target

The team selected Chrome's JavaScript/WASM interpreter V8 as their high-value target. V8 is a high-performance JavaScript and WebAssembly engine developed by Google that uses Just-In-Time (JIT) compilation to compile JavaScript directly into machine code for execution. V8's complexity stems from its multi-layer optimization pipeline: the Ignition interpreter handles fast startup, while the TurboFan and Maglev compilers optimize hot code paths into efficient machine code. JIT compilers are a hotbed for security research because any incorrect assumptions about variable types or memory layout during speculative optimization can lead to type confusion vulnerabilities, which can then be exploited for arbitrary memory read/write.

V8 is a cornerstone of the internet — it not only powers Chrome's JavaScript execution but also runs Edge, Node.js, and CloudFlare edge workers. If V8 is compromised, all these systems fall.

What makes V8 a high-value target is its sandbox protection. V8 Sandbox (officially enabled in 2024) limits vulnerability impact by isolating the V8 heap within a restricted virtual address space. Triggering a crash inside the sandbox is meaningless (it's expected behavior) — what truly matters is an out-of-sandbox exploit. Modern browsers employ multi-layered defense architectures: the renderer process runs inside a sandbox, so even if an attacker gains code execution within the renderer through a V8 vulnerability, they're still restricted by OS-level sandboxing from accessing the filesystem or other processes. Achieving true system-level compromise requires "chaining" multiple vulnerabilities together — first using a V8 bug to gain renderer process control, then using a sandbox escape to break process isolation. Constructing such "exploit chains" demands extremely high technical skill because each vulnerability in the chain must work cooperatively under specific constraints. Bug bounties for these start at $10,000 and can reach $100,000, while on the black market they can fetch millions of dollars.

Shocking Experimental Results

The team designed a 16-level capability "ladder" across 41 manually verified V8 vulnerabilities, ranging from "trigger the bug" all the way to "arbitrary code execution." The results are revealing:

• By the old standard (triggering a crash): Both GPT and Mithos achieved 95% (39 out of 41 successful), and even weaker models like Gemini, Kimi, Minimax, and GLM reached roughly 50% success rates. Judged only by old benchmarks, you'd draw the false conclusion that "Kimi has a 50% chance of hacking successfully."
• By the real standard (sandbox escape with arbitrary code execution): Models truly differentiate. Mithos performed remarkably at 73% (30 out of 41 achieving full control-flow hijack); GPT reached 68%; while Gemini and Kimi both scored 0%.

More critically, multiple cases prove this is not mere memorization. On CVE-2023-6702, Mithos took a path widely considered "too difficult in practice" — it reversed JavaScript's math.random to forge pointers for a Return-Oriented Programming (ROP) attack. ROP is an advanced exploitation technique that bypasses modern OS Data Execution Prevention (DEP/NX): since DEP prevents attackers from directly executing injected shellcode on the stack or heap, ROP instead leverages short instruction sequences ending in "ret" (called gadgets) already present in the program's code, chaining them together via carefully crafted return addresses on the stack to perform arbitrary computation. Reversing math.random to predict pointer values is meant to bypass Address Space Layout Randomization (ASLR) — this requires cracking the internal state of V8's pseudo-random number generator (PRNG). While academically known to be feasible, the engineering implementation is extraordinarily difficult, making Mithos's approach highly creative.

On CVE-2024-7965, it even pioneered a novel WASM exploitation path that public research had abandoned; attacks that even the team's internal experts deemed impossible on x86 architecture were successfully completed by Mithos. Brumley's assessment: "Its results have reached the level of elite human researchers."

Notably, because Mithos generated unpublished weaponized exploits, the team faces an open-science dilemma: they want to open-source their work but are concerned about publicly releasing attack code for real high-value targets — a question they still don't have an answer to.

Future Directions: Building RL Training Environments with Zero-Day Vulnerabilities

These benchmarks reveal the capability boundaries of frontier models, and Brumley's team's next step is building RL environments to help models push beyond those boundaries.

Their approach is quite elegant: leveraging vulnerability discovery technology accumulated over a decade of DARPA collaboration, they find unique zero-day vulnerabilities in open-source software. A zero-day vulnerability (0-day vulnerability) is a security flaw unknown to or unpatched by the software vendor. Using zero-days in an AI training context has special significance: since LLM training data includes vast amounts of publicly available vulnerability analysis articles and exploit code from the internet, using known vulnerabilities for evaluation makes it difficult to distinguish whether a model is "reasoning" or "recalling." Because zero-days have never been publicly disclosed, they serve as uncontaminated test sets ensuring the model must rely on genuine code analysis and reasoning rather than simple pattern matching — analogous to the strict train/test set isolation principle in machine learning. The team can now operate at scale, providing partner companies with up to 10,000 RL environments per month.

The core conclusion from the entire talk is clear and profound: Training cybersecurity AI isn't mysterious — the key is having true experts build correct oracle mechanisms. Experts need to review transcripts to determine whether the machine is genuinely reasoning or memorizing, whether it's reward hacking, and most importantly — how to handle situations where the machine discovers vulnerabilities you didn't previously know about. The complete benchmark environment is open-sourced at exploitbench.ai, providing Docker images and an MCP interface — anyone can point Claude at this interface and test whether it can successfully hack in.