AI Gateway
An AI gateway is a governance layer for large language model invocations, responsible for cross-cutting concerns such as model routing, centralized API key management, and cost and quota control. It abstracts away API differences across LLM providers. Representative implementations include LiteLLM, Portkey, and Kong AI Gateway.
Core Facts
Timeline (last 90 days)
AI gateways need to complete usage statistics, content filtering and interruption handling during the streaming process, which is more complex than the traditional request-response model because the gateway needs to make real-time decisions amid the data flow.
LLM routing is one of the core capabilities of AI gateways, referring to the gateway automatically selecting the most suitable backend model based on request characteristics.
The positioning of AI gateways is expanding from mere traffic forwarding and API key management to taking on model selection and decision-making.
AI网关通常充当反向代理或SDK包装器，其数据流大致为：网关接收入站HTTP请求并创建根span，随后为具体操作创建子span（如`retrieval.search`、`llm.inference`、`llm.generate`），span通过OTLP批量导出到collector，再路由到Jaeger或Tempo进行存储与可视化。
When evaluating a gateway's detection capability, one should prepare three categories of probes—benign, direct injection, and indirect injection—route all of them through the gateway rather than directly to the provider, and measure both attack pass-through rate and legitimate-request false-block rate.
AI gateways will soon become the single component within an organization that can see every prompt, every completion, and every tool call.
An AI gateway controls what content enters and exits the model, whereas an MCP gateway controls what the model is permitted to do after it decides to take an action.
Every gateway interaction should produce a structured record containing trace_id, user_id, timestamp, model name, input/output token counts, latency, policy violations, and a list of tool calls, with the trace ID used to correlate multiple fanned-out calls.
The EU AI Act's high-risk category requires systems to automatically log events sufficient to reconstruct the decision process, making gateway log tamper-resistance and data-residency options a legal compliance requirement.
There are three data-plane deployment options—SaaS, self-hosted, and hybrid—with self-hosting often being the only acceptable approach in the financial, healthcare, and public sectors.
5 more timeline events
All Facts (17)
Modern AI applications adopt a "front-end and back-end separation" architecture, where the front end handles user interaction and engineering capabilities, the back-end foundation handles language understanding and code generation, and the two communicate via a standardized API protocol.
65%Unverified随着大模型生态的碎片化加剧，统一的AI接入层和管理层变得不可或缺
70%UnverifiedAI gateways need to complete usage statistics, content filtering and interruption handling during the streaming process, which is more complex than the traditional request-response model because the gateway needs to make real-time decisions amid the data flow.
50%UnverifiedLLM routing is one of the core capabilities of AI gateways, referring to the gateway automatically selecting the most suitable backend model based on request characteristics.
50%UnverifiedThe positioning of AI gateways is expanding from mere traffic forwarding and API key management to taking on model selection and decision-making.
50%UnverifiedAI网关通常充当反向代理或SDK包装器，其数据流大致为：网关接收入站HTTP请求并创建根span，随后为具体操作创建子span（如`retrieval.search`、`llm.inference`、`llm.generate`），span通过OTLP批量导出到collector，再路由到Jaeger或Tempo进行存储与可视化。
50%UnverifiedThere are three data-plane deployment options—SaaS, self-hosted, and hybrid—with self-hosting often being the only acceptable approach in the financial, healthcare, and public sectors.
50%UnverifiedAI gateways will soon become the single component within an organization that can see every prompt, every completion, and every tool call.
50%UnverifiedAn AI gateway controls what content enters and exits the model, whereas an MCP gateway controls what the model is permitted to do after it decides to take an action.
50%UnverifiedWhen evaluating a gateway's detection capability, one should prepare three categories of probes—benign, direct injection, and indirect injection—route all of them through the gateway rather than directly to the provider, and measure both attack pass-through rate and legitimate-request false-block rate.
50%UnverifiedEvery gateway interaction should produce a structured record containing trace_id, user_id, timestamp, model name, input/output token counts, latency, policy violations, and a list of tool calls, with the trace ID used to correlate multiple fanned-out calls.
50%UnverifiedThe EU AI Act's high-risk category requires systems to automatically log events sufficient to reconstruct the decision process, making gateway log tamper-resistance and data-residency options a legal compliance requirement.
50%UnverifiedAI gateways introduce a Guardrails mechanism that allows enterprises to define policies in natural language, which are then enforced at runtime by a model or a dedicated classifier.
50%UnverifiedAI gateways share the same traffic proxy philosophy as traditional API gateways such as Kong and APISIX, but the detection layer has shifted from rule-based traffic feature analysis to semantic understanding.
50%UnverifiedIntroducing an AI gateway control layer is worth evaluating when dealing with multiple models, multiple providers, cost sensitivity, or compliance requirements.
50%UnverifiedIt is recommended that enterprises adopt a phased AI gateway deployment strategy: Phase 1 pilot for 1–2 months, Phase 2 expansion to 3–5 teams over 2–3 months, and Phase 3 full rollout over 3–6 months.
50%UnverifiedA new product category is taking shape in the industry, commonly referred to as AI Gateway or LLM Firewall.
50%