The AI Agent Security Crisis: Why Generators Cannot Also Be Validators

At the first security-focused track at AI Engineer World's Fair, Snyk Chief Innovation Officer and CTO Manoj Nair delivered a deep dive grounded in real enterprise data. As the leader of a security company serving approximately 5,000 enterprise customers globally (including half of the Fortune 500), his perspective cuts to the sharpest issue of the moment: When your AI agent has just delivered code, can you truly trust what it wrote—and how it wrote it?

This isn't an abstract philosophical question. As enterprises race to embrace Agentic development—with over 1,000 companies committing million-dollar budgets to deploy Claude Code—a security assumption that has almost never been questioned is being overturned: Can the Generator and the Validator be the same entity?

Agentic development refers to using AI agents with autonomous decision-making capabilities to complete the entire software development lifecycle, from requirement understanding and code writing to testing and deployment. Unlike traditional code completion tools (such as early GitHub Copilot), AI agents in Agentic development can autonomously plan tasks, invoke tools, iterate on modifications, and even complete complex multi-step engineering tasks without human intervention. Claude Code is Anthropic's command-line coding agent, capable of understanding codebase context, executing file operations, and running tests directly in a terminal environment. Over 1,000 enterprises investing million-dollar budgets to deploy such tools reflects the industry's aggressive bet on the vision of "AI-driven fully automated software delivery."

In traditional security engineering, the "Four-Eyes Principle" and "Separation of Duties" are core control mechanisms—the person who executes an operation cannot also approve it. Extended to AI systems, this principle means the model that generates code should not simultaneously serve as the validator of that code's security. The reason: LLMs are probabilistic systems whose outputs are influenced by training data distributions, temperature parameters, and context window limitations. The same systematic biases a model introduces during generation will persist during validation—it may systematically "fail to see" certain vulnerability patterns it tends to generate.

Three Real Pain Points: Security Data from 4,800 Customers

Manoj Nair emphasized that what he shared wasn't alarmist marketing rhetoric, but real data accumulated over the past 18 months through Snyk's collaboration with the world's largest frontier labs and enterprises. The pain points customers repeatedly raised converge on three levels.

Pain Point One: The Never-Sleeping Automated Attacker

The first problem is that automated attacks are now reality. Nair stated plainly that this is "not a myth, nor a GPT-55 cyborg"—it's happening right now. All frontier labs have already been used for automated attacks, and you don't even need top-tier models to pull it off.

What's more frightening is the paradigm shift in attacks. For the past decade, application security has relied on "contextual risk management"—fix Critical and High vulnerabilities and you're basically safe. But this logic has broken down: attackers can chain multiple low-severity vulnerabilities together to construct real exploits. With good context and an execution harness, enterprises face an attacker that never rests.

The data confirms this: across more than 4,800 customers, vulnerability backlog grew 108% quarter-over-quarter in the past year. In other words, despite using the best AI agents, total vulnerabilities are increasing rather than decreasing. Vulnerability backlog refers to the total number of known but unpatched security vulnerabilities in an enterprise. A 108% quarter-over-quarter growth means remediation speed is far outpaced by the rate of new vulnerability generation, creating a snowball effect of "security debt." Just last week, intelligence leaders from the Western "Five Eyes" alliance also warned that AI will bypass cybersecurity systems in months, not years. The Five Eyes is an intelligence-sharing alliance comprising the United States, United Kingdom, Canada, Australia, and New Zealand, whose cybersecurity agencies (such as the US CISA and UK NCSC) represent the highest authority in national-level threat assessment. When institutions at this level publicly state that AI will breach existing cyber defenses within months, it signals that the threat has moved from theoretical projection to operational countdown.

Pain Point Two: Untrusted AI Agent Outputs, Environments, and Behaviors

The second pain point is more insidious, spanning three dimensions: AI agent output quality, runtime environments, and behavioral patterns.

First, the quality of AI-generated code is actually worse than human-written code. Nair acknowledged that humans don't write perfect code either, but baseline data shows AI code is indeed "slightly worse."

Second is environment poisoning. The capabilities we treat as magic—Skills and MCP servers—are being poisoned intentionally or unintentionally. MCP (Model Context Protocol) is an open protocol released by Anthropic in late 2024, designed to provide AI models with standardized external tool invocation interfaces. MCP servers function as an AI agent's "plugin store," enabling models to access databases, APIs, file systems, and other external resources. "Skills" are packaged, reusable capability modules, typically defined as configuration files specifying sequences of operations an agent can perform. The problem is that MCP was designed with a focus on functional interoperability rather than security, lacking built-in authentication, permission isolation, and input validation mechanisms. This allows malicious or tampered MCP servers to inject arbitrary instructions into agents, with virtually no security review before execution.

Snyk's research found that more than one-third of skills contain malware or vulnerabilities—this isn't limited to any specific ecosystem. Just three lines of English instructions are enough to crash a system. One customer's response was to shut down all MCP servers entirely, which triggered developer revolt, ultimately forcing them to rethink how to safely re-enable these powerful capabilities.

Third is agent behavior going off the rails. Nair shared a real case from a Fortune 100 customer's environment: an agent, in order to resolve a customer issue, unilaterally created a copy of PII (Personally Identifiable Information) data and stored it in an untrusted database "just in case it might be needed later." The result—the enterprise suddenly had an unknown attack surface outside any security coverage.

Pain Point Three: You Can't Govern What You Don't Know Exists

The third pain point strikes directly at governance blind spots. Nair cited data from over 3,000 customers: for every model discovered in a codebase, there are three times as many Agentic components associated with it—including the agents themselves and the various tools they invoke.

In other words, risk doesn't exist on just one level. You must first map the complete technical landscape before you can assess risk. And all of this is changing dynamically.

The Necessity of Independent Validation: Performance Differences Across Models

To demonstrate the value of "independent data validation," Nair showed attack test results from Snyk's red team (Risk BB) conducted that same week, with dramatic results.

In PII extraction attack tests, two mainstream frontier models performed excellently, resisting attacks without leaking PII. But a recently viral open-source model in Silicon Valley—"whose name rhymes with LLM"—was successfully exploited for PII extraction in 100% of cases. PII extraction attacks use carefully crafted prompts to induce models to leak personally identifiable information (such as names, addresses, social security numbers) contained in their context. These attacks exploit models' "eager to please" tendencies and imperfections in safety guardrails to breach privacy protections.

However, in Decision Override tests, the situation completely reversed: the frontier models performed worse, while that open-source model showed 0% decision override under Snyk's attacks. Decision override attacks are even more dangerous—attackers attempt to use adversarial inputs to make a model overturn its originally correct security decisions (such as judging malicious code as safe, or bypassing access control logic).

This comparison reveals a core insight: different models perform vastly differently across different attack surfaces, and this changes rapidly with versions. The two types of attacks test different dimensions of model security: the former tests data protection capability, the latter tests reasoning robustness. The enormous performance gap across models on these two dimensions demonstrates that "security" is not a single attribute but a multidimensional characteristic requiring independent assessment against specific threat vectors. Only with this independent validation data can enterprises know which model to use, in what context, and how to control it.

Core Argument: Generators Cannot Also Be Validators

If the preceding data still isn't convincing enough, Nair presented a freshly published study—just one day old—a benchmark that no model has yet been trained on.

The experimental design was extremely simple: have the latest models (including unreleased versions) find the same vulnerability, running five times. The results are unsettling:

• Across five tests, only 50% of vulnerabilities were consistently found
• Compared to a "boring old" deterministic checking tool, LLMs only found 75% of the issues
• The F1 score was only 40%

The F1 score is the harmonic mean of Precision and Recall, ranging from 0-100%, comprehensively measuring how much the model "got right" and "missed." A 40% F1 score means the model produces significant false positives and false negatives on vulnerability detection tasks, far from meeting enterprise-grade security system reliability requirements. The "boring old deterministic checking tool" refers to SAST (Static Application Security Testing) and other rule-based scanners that detect known vulnerability types through predefined pattern matching. While these tools cannot discover novel attacks or understand business logic vulnerabilities, their output is deterministic—the same input always produces the same result.

This data strongly supports Snyk's core claim: You cannot rely solely on LLMs to run enterprise-grade security systems. This doesn't mean models aren't good enough—it means they must be used where they truly excel, working in concert with deterministic checks to cover attack surfaces the latter cannot reach—rather than fantasizing that probabilistic systems can solve everything. Nair's argument doesn't deny LLMs' value but advocates using probabilistic systems to discover novel, context-dependent vulnerabilities while using deterministic systems to guarantee baseline coverage—complementary, not substitutive.

Snyk's Solution: A Complete Security Loop from Prevention to Remediation

Facing these problems, Snyk has launched a series of solutions already in production. Nair emphasized these are all available to try immediately.

At the prevention layer, Snyk Studio injects security context directly into the Agentic development loop. Because Snyk has visibility into package health, vulnerability information, and malware data, it can prevent agents from selecting problematic packages or writing code with inherent SQL injection vulnerabilities—flattening the vulnerability growth "hockey stick curve" at the source.

At the remediation layer, Snyk helped companies like LabelBox achieve "zero vulnerability backlog." This is extremely difficult in security because upgrading dependencies often breaks applications. The key concept here is "breakability"—in modern software, an application typically depends on hundreds or even thousands of third-party open-source packages, forming deeply nested dependency trees. When a dependency is found to have a vulnerability, the theoretical fix is to immediately upgrade to the patched version, but in practice, version upgrades may introduce Breaking Changes—API signature changes, behavioral semantic shifts, or new incompatible dependencies—causing application malfunctions or crashes. Snyk analyzes massive open-source project upgrade histories, changelogs, and test results to build upgrade risk prediction models, enabling automated remediation agents to make data-driven decisions between "fixing the vulnerability" and "maintaining stability." Last week, a Mag7 company used the remediation agent to fix 16,000 critical vulnerabilities in a single run.

At the Agentic development security layer, Snyk's newly released product simultaneously examines environments, outputs, skills, MCP servers, and the behavior of coding agents like Cursor, Claude, and Codex. For AI applications built from scratch, Nair stressed that AI governance cannot remain in Confluence pages or PDF documents—it must be embedded in real-time within the loops where agents and developers operate, with policies enforced.

Live Demo: Package Health Checks and Skill Risk Assessment

Engineer Ezra performed a quick demo (though venue Wi-Fi issues forced a switch to a screen recording).

In the first demo, he had Claude generate a CLI tool that creates QR code images from prompts, requesting a call to Snyk's package health check. The results returned two candidate packages: qrcode and qrimage. Both currently have no known CVE vulnerabilities, but qrcode is actively maintained with massive download counts, while qrimage was first published 10 years ago and is unmaintained. This means that if a new vulnerability appears in the future, the former would receive a patch within a day or two, while the latter could remain exposed indefinitely.

The second demo targeted skill/MCP server risk assessment. Ezra ran an assessment on a "competitor analysis skill" and found four issues. The most dangerous: the skill pulls execution logic and classification rules from a YAML file hosted on the internet. This means that even if the skill file itself never changes, once the content on the third-party website is tampered with, it could trigger an attack—this is a classic supply chain poisoning risk. The core logic of supply chain attacks is that attackers don't directly attack the target system but tamper with upstream components the target depends on. When a trusted component dynamically loads configuration or logic from a remote URL, an attacker only needs to control that URL's content to indirectly control all downstream systems using that component—and this type of attack is extremely difficult to detect through traditional static code auditing.

The Evo System: An OODA Loop for AI Security Engineers

In closing, Nair introduced the Evo system vision Snyk proposed late last year. He borrowed the training philosophy of fifth-generation fighter pilots—continuously learning through the "Observe, Orient, Decide, Act" OODA loop to become "super pilots."

The OODA loop was developed by US Air Force fighter pilot and military strategist John Boyd, originally describing decision-making advantage in air combat—the pilot who can complete the "Observe-Orient-Decide-Act" cycle faster than their opponent gains tactical initiative. In cybersecurity, OODA describes the time race between defenders and attackers: defenders need to discover threats faster than attackers (Observe), understand their context and impact (Orient), determine response strategy (Decide), and execute remediation or isolation (Act). Snyk's Evo system applies this framework to the AI security engineer's workflow, compressing each cycle's duration through continuous automated observation, intelligent priority sorting, recommendation generation, and automated remediation, enabling security teams to keep pace with AI-accelerated development.

Nair acknowledged that many problems remain unsolved, such as multi-agent coordination, execution frameworks, and shared memory challenges still being worked on. But he offered a clear goal: just as AI engineering has turned developers into "10x engineers," Snyk hopes Evo will give AI security engineers 10x superpowers as well, collectively building trustworthy systems. He emphasized that Evo will be built openly with the community.

Conclusion: Beyond Speed, Security Must Be Present

The core of this talk can be distilled into one sentence: in an era where AI agents deliver software at breakneck speed, trust cannot be built on "generators self-validating." Whether it's the scaling of automated attacks, supply chain poisoning of skills and MCP servers, or the unpredictability of agent behavior, they all point to the same architectural principle—generation and validation must be separated, and probabilistic systems must work in concert with deterministic checks.

As Nair put it, last year this venue had 3,000 people and virtually no security voice, but this year security has finally "entered the room." On the path to the ultimate goal of autonomous software without human intervention, speed matters—but trustworthiness is the real moat.