LLM Gateways Are Not Control Planes: The Truth About How Agents Bypass Governance

An Overlooked Architectural Problem

Nowadays, you can find an LLM Gateway in almost every AI team's tech stack. It handles model call routing, centralized credential management, logging, rate limiting, and even cost tracking. Tools like LiteLLM, Portkey, and OpenRouter have become standard fixtures in many production environments.

But a technical discussion on Reddit raised a fairly fundamental — yet frequently overlooked — question: What happens if an Agent simply doesn't use the gateway?

The question seems simple, but it cuts to the heart of Agent Governance. When we deploy a gateway in our stack, we implicitly assume it serves as the "governance boundary." The reality, however, is that a gateway can only govern traffic that actually passes through it — it's powerless against traffic that bypasses it.

Traffic Control ≠ Path Control

The original post used a concise architecture diagram to expose the core of the problem:

         ┌──→ LLM Gateway ──→ Models
Agent ───┤
         ├──→ Direct provider API
         ├──→ Direct MCP/tool endpoint
         └──→ Other external egress

In this structure, an Agent can freely choose not to route through the gateway at all. It can:

• Call model provider APIs directly
• Access MCP/tool endpoints directly
• Make requests through other external egress paths

Meanwhile, the gateway keeps "doing its job" — routing, logging, and rate limiting as usual. But it no longer governs this Agent.

This leads to a key insight from the author: traffic control and path control are two entirely different problems. A proxy cannot enforce constraints on traffic that never reaches it. In other words, if your governance mechanism depends on the premise that "traffic must pass through the gateway," then the entire governance framework becomes meaningless the moment an Agent has the ability to bypass it.

The Gateway Is a Component, Not a Boundary

The central argument here is: an LLM gateway should be treated as one component of Agent governance, not the governance boundary itself.

This represents an important mental model shift. Many teams deploy a gateway and develop a false sense of security — believing that as long as traffic flows through the gateway, the Agent is under control. In reality, the governance capability a gateway provides only covers traffic that "chooses" or "is forced" to pass through it. A truly enforceable governance boundary must be established at deeper layers: the network layer, the identity layer, and beyond.

Building a Governance Architecture That Cannot Be Bypassed

To address this problem, the original post proposed a more rigorous layered architecture:

Agent
  ↓
Agent Gateway
  ↓
LLM Gateway / Governed Tools
  ↓
Models + APIs + Network/Egress Enforcement + Identity + Shadow Discovery

The key insight behind this architecture is that it no longer assumes "routing traffic through a gateway equals a governed Agent." Instead, it builds the governance boundary through multiple layers working together:

Egress Enforcement

This is the critical mechanism for blocking bypass paths at the network layer. Even if an Agent has credentials and the intent to bypass the gateway, if the network layer restricts which egress destinations it can reach, it simply cannot connect directly to ungoverned model or tool endpoints. This transforms governance from "opt-in at the application layer" to "mandatory enforcement at the network layer."

Shadow Discovery

Shadow discovery is used to detect covert paths that bypass the gateway. In a distributed Agent environment, it's nearly impossible to guarantee that every Agent will dutifully route through the gateway. Shadow discovery actively identifies these "shadow traffic" flows and "shadow endpoints," exposing and closing bypass paths.

Identity

The identity layer ensures that every call from every Agent has a clear, verifiable attribution. This is the prerequisite for actually enforcing access control policies — without identity, fine-grained authorization and auditing are impossible.

The post highlighted Lyzr Open Controller as particularly interesting in this regard: it pairs gateway functionality with egress enforcement and shadow discovery specifically to detect and close bypass paths, rather than simply assuming "routing equals governance."

Why This Problem Will Only Get More Critical

The author predicts that as agent estates become increasingly distributed, the severity of this problem will continue to grow. Today's Agents may be scattered across:

• Kubernetes clusters
• Cloud agent runtimes
• MCP servers
• Internal self-hosted services

In such a fragmented environment, the assumption that "all traffic will obediently flow through the gateway" becomes harder and harder to hold. Every Agent with credentials and network access is a potential bypass point.

This leads to the sharpest question in the original post: If an Agent has both credentials and network access, and can directly call models or tools, what is actually stopping it from bypassing the gateway?

Conclusion: The Real Answer Beyond the Architecture Diagram

The value of this discussion lies not in providing a standard answer, but in puncturing a widely held architectural illusion. The original author specifically emphasized that what they wanted to hear were approaches that actually work in real production environments — not designs that "should work" on an architecture diagram.

This reminds us that operationalizing Agent governance is far more complex than drawing diagrams. From an engineering practice perspective, several directions are worth thinking through deeply:

• Tighten credential distribution: Does an Agent actually need raw credentials that can directly call provider APIs? Or should it only be issued short-lived tokens proxied through the gateway?
• Network-layer enforcement: Use egress policies, firewall rules, or a service mesh to cut off bypass paths at the network level, turning "use the gateway" from an option into the only option.
• Continuous discovery and auditing: Assume bypasses will happen, and continuously surface them through shadow discovery mechanisms rather than relying on one-time configuration.

Ultimately, a true control plane must be one that Agents cannot bypass. When bypassing becomes possible, the gateway is merely an "optional convenience" rather than a "governance boundary." That distinction is precisely the dividing line between "looking controlled" and "actually being controlled."