Why Token Blocking Cannot Revoke an AI Agent's Tool Permissions

Why "Blocking Wording" Is Not the Same as "Revoking Permissions"

When an AI Agent holds a credential that can trigger a customer data export, simply suppressing phrases like "send externally" in its text output does nothing to revoke that credential. Token-level controls operate at the generation stage, while the real business consequences occur the moment a service accepts and executes a request.

This is a misalignment that's easy to overlook in many security designs. Developers often focus on "making the model not say certain things," while forgetting that the model's language output is fundamentally just a proposal. What actually determines "what can happen" is the capability boundary exposed by the server — not what the model says out loud.

Compliant Wording Can Still Trigger Dangerous Actions

Imagine a synthetic test environment: a direct export function is blocked by a hook, but a generic "file sharing" function can deliver the same data to the same partner. The model can be perfectly compliant with the blocked wording — it never utters the forbidden phrase — yet simply selects an alternative tool and achieves the exact same outcome.

This exposes a core problem: an uncovered capability map. When controls are built around function names or output wording, there will always be side routes left unblocked. The model doesn't need to "speak out of turn" — it just needs to find a functionally equivalent path that isn't covered by policy.

The Fundamental Difference Between Token Control and Action Control

Token control is a generation-time constraint. It can influence how the model speaks, but it cannot restrict what ultimately gets executed. These two things operate on entirely different levels:

• Token layer: Shifts the probability distribution of text output; suppresses specific words and phrases
• Service layer: Determines whether a request is actually executed and what state gets modified

Placing all security responsibility at the generation stage is equivalent to betting that "the model will never find a way to rephrase its way around the rules" — which is not a reliable assumption when the model has access to flexible tool-calling capabilities.

From a technical implementation perspective, token controls are typically realized through system prompt injection, output filter regex matching, or RLHF fine-tuning — all of which act on the language model's inference process. These controls have legitimate uses, such as preventing harmful content generation or enforcing output format consistency. But in agentic scenarios, the model's output is no longer just text for human consumption — it's structured instructions, like JSON-formatted function call parameters, that directly trigger downstream tool calls. Even if a filter successfully blocks a sensitive keyword, the model can still generate semantically equivalent call parameters that bypass literal-match rules. This is precisely why the security community increasingly emphasizes the Principle of Least Privilege in Agent credential design: rather than trying to anticipate and intercept every possible expression, it's far better to ensure the credential itself only permits explicitly authorized operations.

Group by "What Can Be Changed," Not by Function Name

The right approach is to group capabilities according to what an action can change, not what the function is called. If both direct export and generic file sharing can ultimately deliver customer data to an external partner, they belong in the same high-risk category on the capability map and should be subject to the same policy.

In terms of credential design, Agents should be issued a scoped credential explicitly restricted to:

• Tenant
• Recipient
• Operation type

This way, even if the model selects an alternative tool, the credential itself won't allow it to reach unauthorized tenants, recipients, or operation types. The permission boundary no longer depends on whether the model "behaves" — it's hard-enforced by the credential's scope.

Using "Harmless Forbidden Requests" for Penetration Testing

To verify that controls are genuinely effective, send a harmless-but-should-be-blocked request through every possible route, then read the target state to confirm nothing actually changed. This final-state-based validation approach is far more reliable than checking logs for prohibited wording — because it directly answers the only question that matters: was the business state modified?

Enforce Policy at the Last Unavoidable Service Boundary

Output constraints still have value and should be kept where they help. But the real action policy must be enforced at the last unavoidable service boundary.

This boundary is critical because it's where all invocation paths inevitably converge. No matter which side route the model takes or how it packages a request, the request will ultimately arrive at the service that executes it. Performing authentication and policy enforcement there is the only way to truly guarantee that "no matter what the model says, it cannot exceed its permissions."

Implications for AI Agent Security Design

This line of thinking has direct practical guidance for teams building production-grade AI Agents:

• Don't treat token blocking as a permission control mechanism — it constrains expression, not capability
• Shift the focus of permission management from the model layer down to the service layer
• Use scoped credentials as hard boundaries, making privilege escalation physically impossible
• Validate real state with penetration testing rather than trusting output wording

Ultimately, an Agent's words are just a proposal — the service's capabilities determine reality. Anchoring security at the service boundary is the only truly solid defense as AI Agents become increasingly autonomous.

Background

This approach of modeling around "business effect" rather than "interface name" is known in security as Capability-Based Access Control, and it differs from traditional Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC) models. Under RBAC, permissions are often bound to function names or API endpoints, making it easy to introduce vulnerabilities where two differently-named operations have identical effects. Scoped credential practices are more aligned with the spirit of the OAuth 2.0 scope mechanism: each token carries only the minimum permission set needed for the current task, and the semantics of those permissions are defined at the business layer by the authorization server — not interpreted at runtime by the caller. Applying this to AI Agents means that a credential is hard-coded at issuance time with the permitted tenant, recipient, and operation type, so the service never needs to trust any self-declaration made by the Agent.

This testing methodology is conceptually similar to a "Proof of Concept" in penetration testing, but deliberately kept harmless so it can be safely executed in production or production-like staging environments. Its core insight is introducing state assertions into Agent security testing: test cases use "was the target database record, filesystem, or external service state modified?" as the sole assertion — not "did the model say something prohibited?" This is consistent with the philosophy of integration testing in traditional software, simply extending the subject under test from deterministic code to generative Agents. In practice, you can pre-configure a "canary state" for critical business operations — a dedicated observation point that doesn't affect real data — enabling continuous automated penetration testing without relying on manual log review.