Long Documents Don't Work: Why AI Agents Can't Follow Your Rules

An Overlooked Reality: Agents Are Not Obedient Employees

When enterprises begin deploying AI Agents at scale, one intuitive approach is: write a comprehensive policy document (say, Handbook.md), put all rules, boundaries, and expected behaviors into it, and hand it to the Agent to follow. It sounds reasonable—just like giving a new employee a staff handbook.

However, a heated discussion on Hacker News (54 upvotes, 28 comments) pointed out a harsh truth: long policy documents cannot reliably constrain AI Agent behavior. This observation reveals a widely underestimated problem in current Agent engineering practices.

Why Long Documents Fail: Three Core Reasons

Context Dilution Effect

The attention mechanism of Large Language Models (LLMs) does not treat all input content equally. When you stuff a policy manual of thousands or even tens of thousands of words into the context window, the model's attention to content in the middle of the document drops significantly—this is what researchers call the "Lost in the Middle" phenomenon.

This phenomenon has a solid experimental foundation. In 2023, researchers from Stanford University and UC Berkeley published systematic studies proving that when LLMs process long contexts, information retrieval accuracy for content at the beginning and end of documents is significantly higher than for the middle. This is closely related to positional encoding and attention allocation mechanisms in the Transformer architecture—while self-attention can theoretically attend to tokens at any position, in practice, models tend to assign higher weights to content closer to the current generation position. Even the latest models claiming support for 128K or even million-token context windows (such as Claude, Gemini, etc.) have only mitigated rather than fundamentally solved this problem.

In other words, the critical constraint you wrote as Rule #37 in your handbook is likely to be diluted, forgotten, or ignored during the model's actual decision-making. The longer the document, the lower the probability that any single rule will be effectively enforced. This is similar to how humans "skim for key points" when reading handbooks, but a model's "forgetting" is far more unpredictable.

Rule Conflicts and Priority Ambiguity

Another fatal flaw of long documents is that internal consistency is nearly impossible to guarantee. When policy clauses number in the dozens or hundreds, implicit conflicts or priority ambiguities between rules are inevitable. Human managers can resolve such contradictions using common sense and contextual judgment, but an Agent's behavior when facing ambiguous instructions is often unpredictable—it may selectively execute, randomly choose, or even "hallucinate" rules that don't exist in the document.

The root cause of this uncertainty lies in how LLMs handle instruction conflicts. Large language models learn to follow instructions through pre-training and Instruction Tuning, but this compliance is statistical rather than logical. When two rules contradict each other, the model has no built-in formal reasoning engine to determine priority—instead, it makes choices based on statistical probabilities of similar patterns in the training distribution. This explains why Agent behavior in rule-conflict scenarios exhibits nondeterminism—the same input may produce completely different decisions at different inference temperatures or even across different runs at the same temperature. For enterprise scenarios requiring deterministic behavior guarantees, this statistical "compliance" is clearly insufficient.

The Essential Difference Between Soft Constraints and Hard Constraints

The most fundamental issue is: rules written into prompts or documents are inherently "soft constraints." They are merely suggestions to the model, not enforced boundaries. The model has ample "freedom" to deviate from these instructions, especially when encountering unanticipated edge cases.

The distinction between soft and hard constraints here essentially corresponds to the classic classification of advisory control versus mandatory access control in computer security. In traditional operating system security models, Linux's SELinux and Windows' integrity level mechanisms are typical implementations of hard constraints—even if an administrator makes an operational error, system-level policies can prevent privilege escalation. Rules in prompts more closely resemble application-layer "request-suggestion" patterns, where the model can choose to comply or deviate under certain conditions. This is particularly dangerous in adversarial scenarios: jailbreak attacks, through carefully crafted input sequences, have been repeatedly proven to bypass safety instructions in system prompts. This means that no matter how perfectly written your policy document is, a single malicious user input could render all soft constraints meaningless.

More Reliable Agent Governance Paradigms

An important consensus emerged from the discussion: constraining Agents should not rely on persuasion, but on architecture. The following approaches are considered more reliable:

Enforcement at the Tool Layer

Rather than writing "do not delete the production database" in a document, simply don't give the Agent tool permissions to delete the production database in the first place. Push constraints down to the tool interface (tool schema) and permission system level, making dangerous operations physically impossible. This is the hardest guardrail.

Tool-layer constraint implementation is already concretely reflected in current mainstream Agent frameworks. OpenAI's Function Calling mechanism and Anthropic's Tool Use protocol both allow developers to precisely define the set of tools an Agent can invoke and their parameter schemas. By strictly limiting parameter types and value ranges through JSON Schema, Agents are physically unable to construct operation requests beyond predefined boundaries. For example, you can define a database query tool that only accepts SELECT statements, eliminating the possibility of DELETE and DROP operations at the schema level. This approach follows the "Capability-based Security" design philosophy—the system doesn't ask "are you allowed to do this" but rather "do you possess the capability credential to do this." Without the credential, the operation is directly rejected at the API layer, regardless of the Agent's "intent."

Structured Short Rules Replace Long Documents

If rules must be expressed in natural language, short, focused, and structured instructions are far superior to lengthy prose. Decompose complex policies into concise rule sets targeting specific tasks, inject them into context on demand, rather than dumping the entire handbook at once.

The effectiveness of this strategy has been validated in multiple studies. When rules are limited to 5-10 items, each clearly expressed in a single sentence, model compliance rates are significantly higher than when facing hundreds of rules. In practical engineering, this means establishing a dynamic rule routing mechanism: retrieve the most relevant handful of rules from a complete rule repository based on the current task context and inject them into the prompt, rather than statically loading all policies. This approach is similar to how RAG (Retrieval-Augmented Generation) is applied in knowledge management, except the retrieval targets shift from knowledge documents to governance rules.

Runtime Validation and Guard Mechanisms

Add an independent validation layer (guardrail) after each Agent action to perform programmatic checks on outputs. This is equivalent to placing a "compliance auditor" outside the Agent—any action violating hard rules is intercepted without relying on the Agent's own "self-discipline."

Guardrail mechanisms have formed a relatively mature practice ecosystem in industry. Open-source frameworks like NVIDIA's NeMo Guardrails and Guardrails AI provide declarative rule definition languages, allowing developers to perform multi-layer programmatic validation on Agent inputs and outputs. Typical guard layer implementations include: regex matching to detect leakage of sensitive information (such as credit card numbers, API keys), independent small classification models to determine whether outputs violate security policies, and deterministic business logic validation (such as whether transaction amounts exceed preset thresholds, or whether operation targets are on a whitelist). The key point is that these validation layers run outside the Agent's reasoning loop, existing as independent processes or middleware, and therefore are unaffected by the Agent's own "hallucinations" or instruction forgetting—even if the Agent "decides" to execute a violating operation, the guard layer intercepts it before the operation actually takes effect.

Layered Architecture and Responsibility Isolation

Through multi-Agent collaboration or process orchestration, isolate different responsibilities into different execution units, with each unit possessing only the minimum information and permissions necessary to complete its task, thereby reducing the risk of single-point loss of control.

The design philosophy of multi-Agent layered architecture draws from microservices architecture and the Principle of Least Privilege. In practice, this typically manifests as an Orchestrator Agent responsible for task decomposition and flow control, while multiple specialized Worker Agents each possess only the tools and context needed to complete specific subtasks. For example, an Agent responsible for customer service responses doesn't need—and shouldn't have—access to financial systems; an Agent responsible for code generation shouldn't have permissions to deploy to production environments. Frameworks like CrewAI, LangGraph, and AutoGen all provide this multi-Agent orchestration capability. This isolation not only reduces single-point failure risk (even if one Agent is "compromised," its impact is confined within its permission boundaries), but also makes each Agent's context window more concise, indirectly alleviating the aforementioned context dilution problem—because each Agent only needs to receive the few precise rules relevant to its responsibilities.

Deeper Implications for Agent Engineering

The value of this discussion lies in challenging a rather popular "prompt omnipotence" belief—the notion that as long as prompts are detailed and comprehensive enough, models will obediently comply. The reality is that natural language instructions are always probabilistic and bypassable.

Truly robust Agent systems need to migrate critical constraints from the "prompt layer" to the "system layer." This means engineers should shift their focus from "how to write better policy documents" to "how to design security architectures that don't rely on model self-discipline." Specifically, this requires Agent developers to adopt traditional security engineering mindsets: Threat Modeling to identify scenarios where Agents might lose control, Defense in Depth to ensure that failure of any single defensive line doesn't lead to catastrophic consequences, and Fail-Safe principles to ensure systems default to the most conservative behavior under uncertainty.

This shift parallels a classic lesson from software security: you cannot rely on users "behaving" to ensure system security—you must rely on access control, input validation, and the principle of least privilege. Agent governance is heading down the same path. Looking at the history of web security, the principle "never trust user input" only became industry consensus after countless painful SQL injection and XSS attacks. The Agent security field is at a similar early stage—the principle "never trust that models will voluntarily follow rules" may require an equal number of security incidents before it becomes deeply ingrained.

Conclusion

The Handbook.md metaphor reminds us: placing governance hopes in a long document is a beautiful but dangerous illusion. As Agents take on increasingly autonomous decision-making tasks, what we need is not thicker handbooks, but smarter architectural constraints. The best way to constrain an Agent has never been making it "understand" the rules, but making it "unable" to violate them.

This also means Agent engineering is evolving from an "art of prompt engineering" into a "science of system security." Successful Agent deployments in the future will depend on whether teams can weave security constraints into every layer of the system—from tool definitions to permission models, from runtime validation to architectural isolation—rather than merely relying on a carefully crafted Markdown file.