AI Agent Guardrails: Input/Output Checks and Tool-Level Protection

What Are AI Agent Guardrails

According to the OpenAI Agents SDK definition, guardrails are used to perform checks and validation on user input and Agent output. It explicitly distinguishes two types: input guardrails act on the user's initial input, and output guardrails act on the Agent's final output. In addition, there are tool guardrails, which wrap the function tools you write yourself and perform checks before and after each call.

This definition seems simple, but a guardrail actually has three possible actions: allow, modify, or block. Understanding this is key—many teams initially think only of blocking, but rewriting (for example, redaction or rewriting the prompt) is often more practical than hard blocking.

The tripwire mechanism is a key engineering detail in the guardrail design of the OpenAI Agents SDK. It is not an independent technical component, but an exception-handling convention: when a guardrail determines that input or output is non-compliant, the SDK throws a specific exception object (usually carrying the guardrail name, the decision result, and the original data), thereby immediately interrupting the Agent's entire execution chain. The value of this design is that the guardrail's decision logic is naturally decoupled from business logic—you do not need to write if checks inside every tool function; you only need to catch this exception at the outer layer of the Agent to decide the next action, such as returning a preset message, escalating to a human, recording an audit log, or triggering an alert. By contrast, if a guardrail merely "returns a boolean," callers can easily ignore it, and the security boundary becomes meaningless. The cost of the tripwire is that it leans toward "hard interruption"; for scenarios where you want the guardrail to only flag rather than block, additional configuration or a different callback approach is needed.

The Division of Labor Between Input and Output Guardrails

The typical value of input guardrails is "blocking requests that should not come in." The example given in the original video is very concrete: a customer service Agent uses a very smart model, so it is slow and expensive; therefore, a guardrail paired with a fast, cheap model is used to specifically identify whether the user is asking for help with math homework. Once misuse is detected, the guardrail triggers a "tripwire," the entire run is aborted with an exception, and your code can catch and handle that exception.

The design philosophy here is worth noting: use a cheap model for triage and an expensive model for the core task. This is consistent with the "segmented processing" idea mentioned in Episode 4—one model is responsible for answering, and another is responsible for screening. A related Anthropic article points out that this division of labor usually performs better than having a single LLM call do both things at once.

Execution Modes: The Cost of Parallel vs. Blocking

When a guardrail executes directly determines its cost and effectiveness. By default, input guardrails run in parallel with the Agent run; but if the guardrail triggers, the Agent may already have consumed tokens or even executed tool calls. In blocking mode, the guardrail completes before the Agent starts, and once it triggers, the Agent does not run at all.

This trade-off is very real: parallel is faster but may waste money; blocking is more economical but adds latency. For high-cost tool-call scenarios, blocking mode is usually more cost-effective; for low-latency-first conversation scenarios, parallel is more suitable. The original video also mentions that the OpenAI SDK supports "pausing the run and waiting for human approval or rejection" for sensitive tool calls, which is equivalent to upgrading the guardrail from an automated check to a human gate.

Prompt injection is the core threat model for understanding why guardrails are necessary. It refers to an attacker hiding malicious instructions in user input, retrieved documents, or even tool return results, inducing the LLM to deviate from system settings, leak system prompts, or perform unintended operations. Unlike traditional SQL injection, prompt injection has no clear syntactic boundary—in natural language, "ignore previous instructions" and a normal request can blend seamlessly, so it is difficult to intercept exhaustively with rules. This also explains why the article emphasizes screening tool output before acting on it: web pages, files, or API responses returned by tools may also carry injected content, forming indirect prompt injection. Guardrails here act as probabilistic filters, not deterministic firewalls, so combining multiple strategies, reducing Agent permissions, and isolating the execution environment are what constitute a complete defense line.

NVIDIA NeMo Guardrails' Layered Approach

The second toolkit is NVIDIA's NeMo Guardrails, an open-source Python package used to add programmable guardrails, officially called "rails," to LLM-based applications. Its layering is finer than that of the OpenAI SDK:

• Input rails: run before the model
• Retrieval rails: act on retrieved content
• Dialogue rails: act on the dialogue process
• Execution rails: act on tool calls
• Output rails: run after the model

Among these, input and output guardrails are the most common. The original video specifically mentions a fact-checking guardrail aimed at reducing hallucinations, but its effectiveness "depends strongly on whether the model follows its prompt"—this sentence points out the weak spot of guardrails: guardrails themselves are also model-driven, and they are not deterministic security boundaries.

The separate layer for retrieval rails reflects the new attack surface introduced by RAG (retrieval-augmented generation) architectures. In RAG, the context on which the model bases its answer comes from a vector store or search engine; this content may come from user uploads, third-party web pages, or outdated knowledge bases. Once erroneous information or malicious instructions are mixed in, the model will "faithfully" carry them into the answer. The role of retrieval rails is to screen retrieval results for relevance, safety, or factuality before they enter the prompt, for example by filtering low-quality snippets, flagging sensitive sources, or removing suspected injected text. The difference between retrieval rails and input guardrails lies in what is being inspected: input guardrails look at what the user said, while retrieval rails look at what the system "read." In NeMo's layered model, this layer is often overlooked, but for applications with open knowledge bases or uncontrollable content sources, it is often the key link in preventing hallucinations and injection.

Guardrail Costs, Errors, and Testing Recommendations

Guardrails are not free insurance. The Anthropic article warns that for Agents, guardrails bring higher costs and may produce compounding errors, so it recommends extensive testing in sandboxed, isolated environments together with guardrails.

Regarding prompt injection, the CLAUDE documentation recommends using a lightweight model to screen user input and screening tool output before acting on it, while also emphasizing combining multiple strategies rather than relying on a single line of defense. This corroborates NeMo's layered approach—information from multiple sources points to the same conclusion here: a single layer of guardrails is not enough; layering and combination are the norm.

Summary

The core of guardrails is not "adding a filter," but breaking the Agent's input, tool calls, and output into checkable nodes and deciding at each node whether to allow, rewrite, or block. The OpenAI SDK provides three types of guardrails—input, output, and tool—plus a human approval gate; NeMo Guardrails provides a finer layering of rails; and Anthropic reminds us to pay attention to cost and compounding errors. Together, the three point to a practical principle: guardrails should be designed in layers, fully tested in isolated environments, and accepted as something that itself needs to be validated.