Compiled AI Agent Architecture: Why We Chose Fixed Pipelines Over Runtime Decision-Making

A Core Agent Design Question Worth Revisiting

When building AI Agents that perform file operations across Google Drive, SharePoint, Gmail, and Slack, the most critical architectural decision isn't about model capability — it's about whether to let the model make autonomous decisions at runtime, or to compile natural language instructions into a fixed pipeline. This is one of the most important questions in the Agent space right now.

A Reddit developer (from the commercial product team at The Drive AI) shared their choice: rather than letting the Agent roam freely at runtime, they compile the user's natural language instructions once into a fixed processing pipeline. The trade-offs behind this decision offer valuable lessons for any team building production-grade Agents.

Two Fundamentally Different Approaches

Imagine a user writes this instruction:

"When invoices arrive, file them by vendor; for anything over $5,000, ask me first."

There are two ways to handle this.

Runtime Decision-Making (The Mainstream Approach)

The most intuitive approach is to keep the model in the loop at all times. Each time a file arrives, pass both the instruction and the file to the model, let it decide what to do, and have it call the appropriate tools. This approach is flexible, handles edge cases gracefully, and is the direction most Agent frameworks naturally guide you toward.

This architecture is commonly known as the "ReAct" pattern (Reasoning + Acting), proposed by Yao et al. in 2022. The model first reasons about the current state, then decides which tool to call; after the tool returns results, it enters another round of reasoning. Major Agent frameworks like LangChain, LlamaIndex, and AutoGen all adopt this paradigm by default. Its appeal lies in not needing to enumerate all possible paths upfront — the model can dynamically adjust strategy based on intermediate results, making it excel in scenarios with ambiguous requirements or complex branching logic. However, the cost of this flexibility is that every decision point requires a new model call, and the execution path can't be determined before runtime, making it impossible to show users in advance what the system will do.

Compiled Pipeline (The Approach Covered Here)

The alternative is to compile instructions once and generate a fixed processing pipeline: trigger conditions, information extraction, classification, decision rules, and execution actions.

The model still handles "perception" at runtime — reading documents, extracting vendor names and total amounts — but it no longer decides what happens next. That responsibility belongs to the compiled pipeline. The model is demoted from "decision-maker" to "sensor."

This thinking aligns with the traditional software engineering optimization of "replacing dynamic interpretation with static compilation," and closely mirrors the "Workflow Automation" product direction that has gained traction recently. In AI applications, similar designs are called "deterministic Agents" or "structured Agents" — the core idea is to confine LLM usage to sub-tasks with clear boundaries (such as information extraction and intent classification), while handing control flow logic (like conditional branching and action routing) to code or rule engines. This is consistent with Anthropic's Agent design guidance, which recommends "replacing model decisions with code logic wherever possible." The compilation step itself can be understood as "intent parsing": translating a user's natural language rules into a machine-executable structured representation — essentially moving the prompt engineering work from runtime to deployment time.

Why We Chose Compiled Pipelines: Three Hard Reasons

Operations Are Destructive and Irreversible

Renaming and moving other people's files isn't like a chat message you can take back. The author identifies a critical scale problem: a runtime Agent with 97% accuracy is catastrophic at scale. Out of ten thousand files, 3% means three hundred misplaced documents — often not discovered until months later.

Inspectability

Compiled pipelines are auditable. Users can see the complete compiled workflow before it runs and know exactly what will happen. Runtime Agents can't show users their future behavior — you can only describe their instructions and hope they follow them.

Lower Cost

Each file only requires one model call for extraction, rather than running a full reasoning loop per file. In high-throughput scenarios, this cost difference compounds significantly.

The Trade-offs Are Real

The compiled approach isn't without sacrifice, and the author maintains a rare honesty about his choices.

Real loss of flexibility. A runtime Agent can gracefully handle the "this invoice looks weird" situation; a fixed pipeline will either route incorrectly when it hits an edge case or escalate to human review.

Every change requires recompilation. Users will constantly modify rules, and every edit means regenerating and re-validating the entire pipeline.

The compilation step itself is the hardest part. Turning an ambiguous sentence into a pipeline requires resolving ambiguities the user never even thought about:

• What exactly counts as an "invoice"?
• What happens when a document matches two rules simultaneously?
• When three names appear on a page, which one is the "vendor"?

The author admits that most of their difficulty is concentrated here, not at runtime. This is counterintuitive — people often assume runtime execution is the hard part, but when decision-making is moved forward to the compilation phase, ambiguity resolution becomes the real engineering bottleneck.

The Thorniest Open Problem: Confidence Thresholds and Approval Layers

What the author most wants external input on is the design of confidence thresholds and approval mechanisms.

Every file receives a confidence score; when it falls below the threshold, no action is taken — instead, it's held for human approval. How to set this threshold is the hardest ongoing problem:

• Threshold too conservative: The approval queue becomes "manual filing with extra steps," and the value of automation disappears.
• Threshold too permissive: Silent misplacements occur. This is worse than "no automation at all" because users have already stopped checking.

Their current approach is: every workflow starts in "full approval mode," and as trust is established, users are allowed to gradually relax constraints rule by rule. This works, but the author astutely points out — it's a "UX answer" responding to what should be a "technical" problem.

An Unresolved Technical Question

The specific question the author posed to the community is valuable: Can you get a usable, calibrated confidence score directly from LLM classification? Or do you ultimately need to introduce a separate verification pass?

This touches on a known weakness in current LLM applications — the "confidence" output by large models is often uncalibrated, and for irreversible actions, uncalibrated confidence is almost as useful as no confidence at all.

The probability scores (logprobs) output by LLMs are fundamentally different from truly "calibrated confidence." Calibration means: when a model says a prediction has 80% confidence, that prediction should actually be correct 80% of the time in practice. Research shows that large language models commonly suffer from overconfidence or a mismatch between confidence and accuracy, especially on out-of-distribution inputs. Common engineering mitigations include: temperature scaling to post-calibrate logprobs, introducing an independent verification model to score initial classification results, or estimating reliability by sampling multiple times and measuring consistency. But all of these approaches add latency and cost, and none works universally across all tasks — this is exactly the engineering reality the author refers to when saying "uncalibrated confidence is almost as useful as no confidence at all."

The Broader Significance of This Debate

This case transcends the architectural details of a single product, pointing toward a fundamental design philosophy divide in production-grade Agents:

In scenarios where actions are reversible and fault-tolerant (like chat or draft generation), the flexibility of runtime autonomous Agents is an advantage. But in scenarios where actions are destructive, operate at scale, and errors are hard to detect (like file system operations or financial archiving), inspectable and predictable compiled pipelines may be the more responsible choice.

In other words, an Agent's "degree of autonomy" shouldn't be a technical bragging point — it should be an engineering decision dynamically adjusted based on the consequences of its actions. When the cost is three hundred misplaced documents that someone won't discover for months, trading some flexibility for determinism and inspectability is a completely reasonable trade-off.