Why AI Agents Need a Decision Model — Stop Letting Generative LLMs Do Everything

An Expensive Hidden Assumption

Most Agent frameworks hide a costly assumption: treating the language model as the only hammer in the toolbox. Whenever an Agent needs to make a choice — routing a message, picking a tool, deciding whether to compress conversation history, verifying whether an answer is grounded in facts — the framework's default move is always the same: format a prompt, call a generative LLM, and wait for tokens to stream back.

Treating every decision as a text generation problem is slow, brittle, and wasteful. Generative models predict tokens one at a time over an open vocabulary of tens of thousands of candidates. When you ask one to choose between two actions, it doesn't simply flip a switch — it spends cycles outputting syntax, formatting curly braces, escaping strings, and often tacking on unrequested commentary before getting to the point. You pay for the latency to the first token, you pay for generation delay, and you burn extra compute just to validate whether the output is even valid JSON.

This technical post from Reddit proposes a more sensible architectural pattern: separate generation work from decision work. The critical choices made at the Agent harness layer — the execution framework that hosts and runs the Agent — should be handled by a dedicated "decision model," not a general-purpose generative LLM.

What Is a Decision Model?

A decision model doesn't produce a free-form token stream. Instead, it evaluates the current state against a bounded hypothesis space. In practice, an Agent harness asks three types of questions:

Three Decision Types

• Choice: Select exactly one option from a predefined set — essentially an enum or literal union type.
• Truth: Given the current state, evaluate the probability or truth value of a specific assertion, returning a continuous score between 0 and 1, or a calibrated boolean confidence.
• Score: Evaluate and rank candidates according to a scoring rubric.

The key insight is that the output space is strictly bounded before the call begins. The model doesn't need to generate any syntax or open-ended prose — it directly scores candidates. The payoff: fast execution, low cost, deterministic results, and the elimination of parsing errors, JSON schema violations, and the sycophantic drift that generative models exhibit when "self-censoring."

From an implementation standpoint, decision models can be realized through several paths. One common approach is Constrained Decoding: at inference time, force the model to sample only from a predefined token set, fundamentally closing off open-ended generation with near-zero latency overhead. Another option is a fine-tuned small classification model (such as a discriminative model based on BERT or similar architectures) — these models don't perform autoregressive generation in the first place, making them naturally suited to outputting probability distributions or class labels. A lightweight third option is to use an existing LLM's logprobs interface: request only the log probabilities of specific candidate tokens rather than a full completion. Many inference APIs (such as OpenAI's logprobs parameter) support this pattern, achieving a similar effect without swapping out the model. Each path involves tradeoffs in latency, cost, and engineering complexity — but they share a common principle: output space constraints are enforced at the architecture level, rather than relying on prompt wording to "pray" that the model self-enforces the format.

Four Ways the Decision Model Lands in Practice

The original author describes in detail how this "decision/generation separation" philosophy runs throughout their entire harness.

1. Dynamic Skill Hydration

A common failure mode in production Agents is prompt bloat. Developers want a bot that handles coding, calendar management, database queries, and document retrieval all at once, and the path of least resistance is to dump every skill, prompt rule, and tool schema into the system prompt at the start of every turn.

But this poisons the model's attention. When a model's context is stuffed with forty tools and fifteen pages of instructions, tool selection accuracy degrades and latency spikes.

Their solution is to use the decision model to evaluate candidate skills one by one: for each available skill, the decision model answers a simple truth question — does the current turn or user request actually require this capability? Only skills that score above a confidence threshold get injected into the Agent context for that turn. The system prompt stays lean, and skills unload cleanly when the user changes topics.

This problem is widely known in the research and engineering community as the "Lost in the Middle" effect: studies (including the Stanford 2023 paper of the same name) found that when context grows long, language models utilize information in the middle of the sequence far less than information at the beginning or end. In Agent scenarios, this means even if all tool definitions appear in the prompt, the model may systematically overlook options listed in the middle when actually selecting tools. Dynamic skill hydration sidesteps this problem at the architecture level — by injecting only the few skills truly relevant to the current turn, every injected tool definition lands within the model's effective attention range rather than being buried among dozens of parallel entries. This also explains why accuracy improvements from this pattern aren't just about token savings — they come from improved attention distribution as well.

2. Selective Memory Compaction

Autonomous Agents generate enormous volumes of tool execution traces. A single turn involving bash commands, file searches, or database queries can push thousands of tokens of raw stdout, stack traces, and JSON payloads into the history.

This creates a dilemma: keeping everything risks blowing out the context window and breaking prompt cache hit rates, while aggressively summarizing on a schedule can erase the intermediate line numbers, file paths, or compiler errors that the user or Agent is actively debugging.

The decision model intervenes in a clever way: before a turn executes, the harness asks it whether certain past execution fragments are "settled" — has the subtask in that fragment completed, and does the current user request no longer depend on inspecting that raw stdout or error trace? If confirmed settled, the harness folds those intermediate tool calls and outputs into a compact semantic summary. If the user is still asking about an error from that turn, the raw logs remain untouched. Context stays clean without disrupting active working memory.

3. Hallucination and Safety Gate

When an Agent produces a response — especially in real-time messaging environments like WhatsApp or Telegram — relying on the generative model to fact-check its own output is unreliable. Generative models are inherently inclined to defend their own completions.

So before a message leaves the harness, an independent auditor runs a decision model check. The decision model receives the user query, verified facts from tool outputs, and the bot's proposed reply, then evaluates a boolean condition: is this response an off-topic hallucination, or does it fabricate assertions that contradict the verified facts?

If flagged, the harness aborts the send, injects a targeted corrective thought into the Agent's internal history, and retries. Because the decision model is evaluating a discrete boundary rather than getting drawn into a conversational debate, the latency overhead of this gate is minimal.

It's worth noting that this mechanism is structurally a variant of the Judge Model pattern — using an independent model to evaluate another model's output, rather than asking the generative model to self-reflect. Research shows that when the same model evaluates the truthfulness of its own output, systematic self-consistency bias emerges: the model tends to defend content it has already generated, even when that content contains factual errors. Delegating verification to a structurally independent decision model breaks this feedback loop. From an engineering standpoint, this also means the two models can be optimized for different objectives — the generation model pursues fluency and coverage, while the verification model can be fine-tuned or prompt-engineered specifically for factual consistency tasks. The separation of responsibilities delivers not just latency benefits, but also greater auditability.

4. Hybrid Schema Splitting

Many workflows require structured data containing both discrete flags and free-form descriptions. A ticket classification schema, for example, might include an isUrgent boolean, a category enum, and a summary string. The conventional approach is to hand the entire schema to a generative model and pray the JSON parses.

This harness instead inspects the schema at runtime:

• If the schema contains only discrete types (booleans, enums, literals, or literal unions), it bypasses generative completion entirely and routes directly to the decision model.
• If it's a mixed schema, the harness automatically splits it into two parts: discrete fields go to the decision model, and free-text fields go to the generative model.

Both execute concurrently, and the harness merges the validated results back into a single typed object. The decision model handles classification deterministically; the generative model focuses on synthesizing natural language.

Use the Right Tool: Don't Start a Web Server to Do Addition

The core insight of this argument is clear: generative models excel at synthesizing language, interpreting nuanced concepts, and writing code — but for the deterministic plumbing work inside an Agent harness, they perform surprisingly poorly.

The author offers a precise analogy: using a generative model to decide whether to activate a skill, compact an execution log, or validate output is like spinning up an entire web server just to add two numbers — it works, but it introduces unnecessary latency, cost, and fragility.

Decoupling "decision" from "generation" enables smaller context windows, lower latency, and more predictable behavior. For developers building production-grade Agent systems, this offers an architectural pattern worth taking seriously: don't outsource every judgment to a general-purpose large model. Instead, identify the bounded problems that are fundamentally about classification, evaluation, or scoring — and hand them to deterministic components better suited for the job.

One caveat: the content in this article comes from a single Reddit technical post. The original does not elaborate on whether the "decision model" is implemented via fine-tuned small models, constrained decoding, or other techniques, so readers will need to validate the approach against their own production scenarios.