DecisionLoop: Replacing LLMs with Small Decision Models for Agent Orchestration

At a time when Agent frameworks almost universally default to "driving everything with a large model," a developer on Reddit has proposed a different approach: hand off decision-making tasks like intent recognition, tool selection, parameter generation, and flow control to a small decision model, rather than having the LLM reason through everything from scratch each time. This Python framework, called DecisionLoop, attempts to answer an engineering question—does an Agent's "brain" really need to be that big?

Core Thesis: Decoupling the Decision Layer from the Language Layer

DecisionLoop's basic architecture splits an Agent's work into two layers. The language layer handles understanding and generating natural language, while the decision layer handles "what to do next"—detecting user intent, selecting the appropriate tool, filling in parameters, and determining the flow direction. The framework's innovation lies in the latter being handled by a dedicated small decision model, rather than having the LLM simultaneously perform language understanding and decision reasoning in every conversation turn.

This design brings direct engineering benefits. According to the author, the decision model is small in size with fixed inference paths, so in the tool-calling step, both latency and cost are lower than the "LLM re-evaluates at every step" approach. For scenarios requiring frequent external tool calls, this means potentially saving multiple large model requests within a single conversation.

Small decision models typically refer to language models with parameters ranging from hundreds of millions to a few billion (such as 1B–3B), as opposed to general-purpose large models that often exceed 70B+. Their core advantages are short inference paths, low VRAM usage, and low per-call cost, making them suitable for structured tasks like classification, intent discrimination, and slot filling. For training and fine-tuning small decision models, common approaches include supervised fine-tuning with domain-labeled data, or parameter-efficient methods like LoRA to adapt to specific business processes with limited samples. Due to their small parameter count, they are more sensitive to changes in prompt format and tool descriptions, and their ability to generalize to unseen intents is weaker than large models—which is precisely why the article notes that "switching to smaller models causes noticeable result fluctuations."

In the Agent context, Tool Calling refers to a model selecting the appropriate tool from a predefined toolset based on user requests, generating parameters that conform to a Schema, then executing external APIs and integrating results. It typically involves three sub-steps: tool selection, parameter generation, and result parsing. The traditional approach has the LLM complete all three steps in every conversation turn; DecisionLoop delegates the first two to the decision model, leaving only the parts requiring natural language understanding and generation to the LLM.

Three Test Scenarios and Preliminary Results

The author validated this across three business scenarios: order customer service, flight service desk, and banking service desk. What these three scenarios share is relatively clear processes and limited toolsets, while also needing to handle the diversity of user expression—making them a typical environment where decision models can leverage their structural advantages.

Early test results show that DecisionLoop outperforms an LLM Agent built on LangChain using a 12B model in tool-calling accuracy, while also offering advantages in cost and latency. The author also notes that switching to smaller models causes noticeable result fluctuations, indicating that the decision model's own capability floor directly affects overall performance.

It should be emphasized that the author himself calls these results preliminary observations rather than formal benchmark tests. The test scenarios, sample sizes, and evaluation criteria have not been made public, so these numbers are better understood as "directional signals" rather than directly comparable performance conclusions.

LangChain is currently the mainstream framework for Agent and LLM application development, providing abstractions such as tool wrapping, memory management, and chain orchestration. LLM Agents built on LangChain typically adopt a ReAct-style loop: the LLM outputs a "thought—action—observation" sequence, gradually deciding which tool to call. A typical problem with this approach is that each decision round requires a large model call, with latency and token consumption growing linearly with conversation turns. The article mentions using a 12B model LangChain Agent as a baseline, but it's worth noting that 12B is already considered medium-to-small by today's standards—production Agents in the industry commonly use 70B-class or closed-source large models—so the "ceiling" of this comparison isn't very high, which explains why the author emphasizes the need for controlled comparisons with stronger models.

The Applicable Boundaries of Small Decision Models

The value of this approach depends on the degree of determinism in the task itself. When the toolset is fixed, business processes are enumerable, and user intents can be categorized into a finite set of classes, a decision model can cover most cases with very few parameters, freeing the LLM from repetitive "option judgment."

But the boundaries are also clear. Once a conversation enters ambiguous, polysemous territory requiring substantial world knowledge to navigate, the decision model may lack sufficient semantic understanding to make correct choices—which is why the author actively solicits "difficult conversations to test" in the post. Which conversation types will break through the decision model's assumptions remains an open question.

Another implicit prerequisite is the stability of the toolset and processes. If tools change frequently and processes are constantly adjusted, the maintenance cost of the decision model may offset its inference-time savings.

Feedback the Author Is Seeking

This post is essentially an open-ended architecture discussion. The author lists several specific questions: Is the architecture design reasonable? What real-world scenarios are worth testing? What conversation types might cause the decision model to fail? And how to fairly compare with stronger LLMs?

The last point is particularly critical. Using a 12B model as a baseline has limited persuasiveness, since mainstream Agents in the industry already use larger, stronger models. To prove that DecisionLoop's approach has universal value, controlled comparisons against strong models under identical toolsets and task distributions are needed, with explicit reporting of failure cases—not just average accuracy and cost figures.

The author also mentions plans to build a multi-Agent architecture on top of DecisionLoop, with a routing layer dispatching requests to domain-expert Agents. This direction is consistent with the decision model philosophy: routing itself is a decision problem, and can similarly be handled by a small model.

Controlled Comparison refers to comparing two approaches under conditions where variables are as consistent as possible. Here, at minimum, the following must be fixed: the same toolset and tool descriptions, the same task distribution and sample size, the same evaluation criteria (e.g., whether tool calls are correct, whether parameters are complete, end-to-end success rate), and the same deployment environment (hardware, concurrency). Without these controls, differences in accuracy or latency may stem from task difficulty, prompt differences, or model scale rather than the architecture itself. Furthermore, reporting only averages conceals long-tail failures—in Agent scenarios, a small number of complex conversations often determine whether a product can ship, so reporting the failure case distribution (e.g., stratified by intent category, conversation turns) is more convincing than a single average.

Why This Direction Is Worth Watching

Regardless of whether DecisionLoop itself succeeds, the questions it raises are valuable: in current Agent frameworks, LLMs bear too much decision-making burden that could be handled structurally. Separating "which tool to choose" from "how to express it" can theoretically make the system more controllable in terms of cost and latency, while also making each layer more debuggable.

For teams building production-grade Agents, this offers a comparison approach worth experimenting with: first use a small decision model to handle enumerable decisions, and only call the LLM when genuine language understanding and open-ended reasoning are needed. Where exactly this dividing line should be drawn may require more public experiments like this one to answer.

One-line Summary

DecisionLoop replaces LLM with small decision models for Agent decision-making, preliminarily outperforming a 12B baseline but still awaiting formal validation.

Paragraph Summary

DecisionLoop is a Python framework that advocates decoupling an Agent's decision layer (intent recognition, tool selection, parameter generation, flow control) from its language layer, with a dedicated small decision model handling decision tasks rather than having the LLM reason from scratch each turn. The author conducted early tests in three scenarios—order customer service, flight service desk, and banking service desk—showing better tool-calling accuracy than a LangChain-based 12B LLM Agent, with lower cost and latency. However, the author explicitly labels these as preliminary observations rather than formal benchmarks, with no test details disclosed, and noticeable fluctuations when switching to smaller models. The article notes that the approach's applicability depends on task determinism: advantages are clear when the toolset is fixed and processes are enumerable, but it may fail with ambiguous, polysemous conversations. The author plans to build a multi-Agent routing architecture next.