AI Agent Security: Why Reviewing Explanations Isn't Enough — You Must Constrain the Action Space

The Core Problem: A Valid Explanation Doesn't Mean a Valid Operation

When building AI agent systems, a common assumption is that if we can review the agent's final explanation, we can ensure its behavior is reliable. However, research on AQuA exposes a fatal blind spot in this assumption — an evaluator can approve a seemingly sound explanation while completely missing the erroneous intermediate operations that produced it.

In other words, scoring an agent's final explanation and constraining the actual operations that generated that explanation are two entirely different things. The former is a post-hoc review; the latter is an upfront constraint. When we rely solely on the former, dangerous operations may already have quietly slipped into the system.

A Real AQuA Case: A Future-Data Leak That Passed Review

Appendix B of the AQuA paper documents a highly convincing case. The system once contained an early feature whose explanation sounded causally coherent and logically organized — and it successfully passed a rigorous reviewer's scrutiny.

The problem, however, was this: when calculating "total daily volume" as the denominator, the feature used future information. This is a classic case of data leakage — the model effectively peeked at data that wouldn't exist until after the point in time it was trying to predict.

This type of error is particularly insidious for three reasons:

• Its explanation was entirely reasonable at the natural language level — it was internally consistent
• Reviewers faced with a clearly articulated causal chain found it nearly impossible to detect the hidden temporal violation buried in the underlying normalization operation
• The error occurred at the level of intermediate operations, not in the final output's wording

This is precisely the core contradiction: the reviewer approved the explanation, but missed the erroneous intermediate operation.

Two Defensive Approaches: Post-Hoc Tracing vs. Upfront Constraints

Faced with this problem, the solution AQuA proposes offers important architectural insights.

Natural Language Review: A Post-Hoc Tracing Approach

The traditional approach has reviewers read the agent's explanation and verify whether its claimed logic holds up. But this is fundamentally a check that happens after an operation has already been proposed. It examines a claim, not the operation itself. Any erroneous operation that's sufficiently good at justifying itself has a chance of slipping through.

Constraining the Operation Language: An Upfront Restriction Approach

AQuA's later design took a different path: replacing the original open construction space with a fixed registry of causal operators.

The key effect of this change was that the normalizer using future information became simply inexpressible at the language level. When an erroneous operation doesn't exist in the operation language to begin with, the agent has no way to choose it.

The essential difference between the two approaches can be summed up in one sentence:

Natural language review checks a claim after an action has been proposed; a constrained operator language removes certain actions from the space of possibilities entirely.

The Limitations of Constrained Operation Languages

You may not have noticed, but the AQuA paper doesn't advertise constrained operation languages as a silver bullet. The paper explicitly acknowledges that this approach cannot prevent all possible forms of leakage.

Narrowing the operation space can only eliminate error patterns that have been explicitly excluded from the operator registry. For incorrect combinations of operators within the registry, or leakage pathways the designers never anticipated, a constrained language remains powerless. Constraining the operation space should therefore be understood as a structural line of defense — not an ultimate guarantee.

This honest acknowledgment reminds us that no single defensive mechanism is sufficient to cover all risks in an agent system.

The Core Debate: What Should Be the First Line of Defense in an Agent Pipeline?

This research raises a question worth serious reflection from the AI engineering community:

For an agent pipeline, should the first line of defense be better tracing and evaluation, or a smaller, less flexible operation language?

These two orientations represent different engineering philosophies:

The Case for Tracing and Evaluation

• Preserve flexibility in the operation space, allowing agents to handle more complex and diverse tasks
• Catch errors through robust observability, log tracing, and evaluation mechanisms
• Risk: As the AQuA case demonstrates, post-hoc evaluation has structural blind spots — errors that are good at "explaining themselves" may escape review

The Case for Constraining Operations

• Proactively sacrifice some flexibility, using a restricted set of operators to fundamentally eliminate entire categories of errors
• Make "impossible operations" truly inexpressible, eliminating risk at the design level
• Risk: May limit the system's expressive power, and cannot cover all forms of leakage

Implications for AI Engineering Practice

The value of this discussion extends far beyond AQuA itself. As large-model agents see increasingly widespread deployment in high-stakes domains like finance, healthcare, and quantitative trading, the cost of data leakage and hidden errors grows ever higher.

From an architectural design perspective, a viable best practice is to combine both approaches: use a constrained operation language as the first structural line of defense to eliminate known, foreseeable error patterns; while simultaneously maintaining robust tracing and evaluation mechanisms as a second line of defense to catch unknown errors.

The real lesson is this: don't over-rely on reviewing final outputs. When we can make errors "inexpressible" at the design level, we don't need to expend effort "discovering" them after the fact. Sometimes, the best defense isn't a smarter evaluator — it's a more constrained operation language.