Why AI Agent Security Fails: A Deep Dive into Four Failure Paths

Title

Why AI Agent Security Fails: A Deep Dive into Four Failure Paths

Meta Description

Token Security's Kayla Williams breaks down the four failure paths of AI Agent security: insufficient coverage, mismeasurement, agent invisibility, and human factors — introducing the core framework: Agent Risk = Probability of Failure × Agent Reach.

Article Content

AI Agents are rapidly entering enterprise systems, but security defenses are lagging far behind. In a technical talk, Token Security's Kayla Williams systematically deconstructed the root causes of AI Agent security failures. Her core argument: don't simply blame security failures on a lack of controls. In probabilistic systems, even the strongest controls will fail at sufficient scale.

Core Formula: Agent Risk = Probability of Failure × Agent Reach

This formula is the foundation of the entire analysis. A prompt injection filter that blocks 99.9% of malicious requests sounds strong, but when daily request volume reaches 10,000, 10 will still slip through. What the Agent does at that point depends entirely on its reach.

Controls reduce the probability of failure, while access controls limit reach. Both are indispensable. This logic is nothing new in traditional security, but when Agents gain autonomous decision-making and multi-step execution capabilities, the magnitude of reach expands dramatically, and risk exposure spirals out of control.

The Action Chain: Where Are the Control Points?

Williams breaks down the Agent execution process into a clear action chain:

• The user issues an instruction
• The system assembles context around the instruction
• The model forms a plan and selects tools
• Actions are executed using credentials
• Operation records are generated

Controls can be deployed at each stage: prompt filters, provenance labels, tool allowlists, permission scopes, gateways, and logging. But the problem is that some failures don't happen within a single step — they happen between steps, usually because context is lost.

Why Context Determines Success or Failure

Judging intent requires context, and judging intent is precisely the core task of probabilistic controls. Anomalies themselves are context-dependent: the more context other systems have about Agent behavior, the lower the misjudgment rate. But it will never be zero.

This is why certain stages require deterministic controls: Is the tool not in the catalog? It can't be used. Does the token lack a permission scope? It can't be used. There's no room for judgment here.

In real-world Agent scenarios, deterministic and probabilistic controls need to work together. One reduces the number of times an Agent is compromised or goes rogue, while the other sets an upper bound on risk.

Prompt Injection is key background for understanding this article. It refers to attackers embedding malicious instructions through carefully crafted inputs (user inputs, web content, documents, tool return values, etc.) to hijack the LLM's execution direction. Unlike traditional SQL injection, prompt injection has no reliable "escaping" mechanism — because in natural language, instructions and data are fundamentally the same medium, and the model cannot architecturally distinguish "this is content to execute" from "this is content to reference." Common variants include direct injection (the user directly issues instructions) and indirect injection (poisoning external data sources the Agent reads). Defenses mainly rely on classifiers, instruction hierarchies, provenance labels, and the like, but they are invariably probabilistic: the model can be bypassed by novel phrasing, and classifiers can misjudge legitimate inputs. This is precisely the technical root of the "structural asymmetry" emphasized in the article — defenders face an open, infinitely generatable attack surface.

Four Failure Paths

Williams categorizes genuine security failures — both attacks that penetrate active defenses and attacks that completely bypass them — into four paths.

Path One: Correctly Designed but Insufficient Coverage

The system operates exactly as designed, but the design's coverage is insufficient. Prompt filtering is a typical case. The problem isn't just their probabilistic nature — there's also a structural asymmetry: attackers have a target they can probe repeatedly with no built-in time constraints, while defenders must account for latency on legitimate prompts. Even when everything runs perfectly, defenders are at a disadvantage.

Tool allowlists are the same — they may be deterministic, but coverage is predefined. Someone secretly installed their own MCP server? Not covered. Someone modified the description of an allowed tool to poison instructions? Not covered either.

Path Two: Measuring the Wrong Thing

Improper baseline setting is a typical problem. If you set a single rate limit for 10 Agents performing different tasks, or invoke the adjudicator on a per-call basis, you will inevitably miss suspicious Agent behavior flows.

Using permissions as a baseline doesn't help either, because a rogue Agent always operates within its grant.

In all cases, lack of context is the root cause of incomplete measurement.

Path Three: Not Knowing the Agent Exists

This is the most common failure path. If you don't know an Agent exists, you can't see that it holds credentials it shouldn't — a service account, permissions from an unapproved built-in tool, or an overly broad OAuth authorization.

Credential management problems are especially prominent: an Agent is deleted, but the OAuth authorization it used often remains alive, because the provisioning process bypasses the identity provider.

Service Accounts and OAuth scopes are the two core concepts in this path. A service account is a machine account for non-human identities, typically granted long-lived keys for service-to-service calls; the problem is that it often has excessive permissions, is difficult to rotate, and isn't bound to a specific person. Once an Agent obtains it, attackers gain a "ghost identity" with no MFA, no behavioral baseline, and the ability to lurk long-term. OAuth scopes define what an access token can do; ideally they should follow the principle of least privilege, but in practice many APIs design scope granularity around the entire system (e.g., "read all emails") rather than specific tasks (e.g., "read a particular email"), forcing Agents to request overly broad permissions. On top of that, Agent provisioning processes often bypass the identity provider (IdP), so authorization tokens survive after the Agent is deleted, creating "orphan credentials" — the most easily overlooked blind spot in audits.

Path Four: Human Factors

Ideally, every irreversible operation should have human review. But this simply doesn't scale. If an approval gate fires 30 times a day, it won't be long before all requests get rubber-stamped through.

Credentials: The Only Context-Independent Variable

After an Agent receives an instruction, tracing the instruction's source and the true operator behind every action is often impossible. In this context-deficient predicament, credentials are the only context-independent element — they either allow access or they don't.

Credentials themselves don't belong to the Agent; they belong to the systems the Agent accesses. When an Agent goes wrong, credentials ultimately determine how far the damage can spread.

But credentials are also a double-edged sword. If the target system only offers admin-level API access, you're forced into an all-or-nothing decision. OAuth scopes need to be tailored to the system rather than the task, so they often have to be set broadly. On top of that, there's currently no widely accepted delegation standard, making it very complex to build credentials scoped "by requester rather than by tool."

The absence of delegation standards is the institutional reason fine-grained authorization is difficult to implement. In human user scenarios, OAuth 2.0 provides authorization codes, PKCE, Token Exchange (RFC 8693), and other mechanisms to express "user A authorizes application B to access resource C on their behalf," but for Agent delegation chains — where "a user authorizes an Agent to call a tool, which then calls downstream services" — there is currently no widely accepted end-to-end standard. The result is that every hop easily loses the original operator's identity, and downstream systems ultimately see only a broad service identity. Directions the industry is exploring include: On-Behalf-Of tokens, verifiable credentials (VCs), and dedicated identity frameworks for Agents. Until these standards mature, teams can only use short-lived tokens, request-level auditing, credential proxies, and other stopgap measures — none of which address the root problem of overly broad authorization caused by scoping "by system rather than by task."

Pragmatic Action Recommendations

Williams' conclusion isn't complicated: examine your systems, ask what the coverage boundaries are for probabilistic and deterministic controls, think about how they'll behave as you scale, and then start closing the gaps.

There are no simple solutions, but inaction is absolutely not an option. For teams deploying AI Agents, these four failure paths offer a practical audit framework — not asking "do I have controls?" but asking "on which path will my controls fail?"

One-line Summary

Agent risk equals probability of failure times reach; controls cannot eliminate risk, only access controls can limit the boundary of loss.

Paragraph Summary

Token Security's Kayla Williams systematically breaks down the root causes of AI Agent security failures, arguing that security failures should not be simply blamed on a lack of controls. She proposes the core formula: Agent Risk = Probability of Failure × Agent Reach — controls reduce probability, access controls limit reach, and both are indispensable. The article breaks Agent execution into an action chain, emphasizing that some failures occur between steps rather than within them, rooted in lost context. From this, she derives four failure paths: insufficient design coverage, mismeasurement, Agent invisibility, and human factors. Williams argues that credentials are the only context-independent element, determining the upper bound of damage spread, but OAuth scopes tailored to systems lead to overly broad permissions, and the lack of delegation standards makes fine-grained authorization difficult. The conclusion is that teams should examine the coverage boundaries of probabilistic and deterministic controls and ask "on which path will my controls fail?"