Put the Decision Model Inside the Agent Loop, Not Beside It

Title

Put the Decision Model Inside the Agent Loop, Not Beside It

Meta Description

Check costs in the agent loop are eating into token budgets. This article explains how a Decision Model returns probabilities in a single forward pass instead of full generative judgment, making safety gates, relevance scoring, and step-by-step validation cheap and affordable, and introduces the S1MB leaderboard and the Darwin-27B-ZTC-v9 benchmark methodology.

Article Content

Build an agent, then trace where the tokens go, and you'll find a surprising proportion: a large share of consumption doesn't come from executing the task itself, but from the agent repeatedly asking itself "small questions."

The Overlooked Cost of Checks

Every small judgment—is this tool call safe? Does this document answer the question? Should this be escalated to a human? Is the output format correct? Did the previous step actually succeed?—is essentially just a yes/no, a single choice, or a score. But under current architectures, every judgment means a full generation process: input the prompt, output reasoning, parse the text, and pray the format didn't drift.

When the frequency of these judgments reaches thousands per minute, the cost of checking exceeds the task itself. So teams start cutting back: disabling safety gates, skipping retrieval relevance tests, letting agents run unprotected. It's not that they think these checks are unnecessary—it's that they're too expensive to afford.

Decision Models: Models That Don't Write

Decision models offer a different form of solution: given a situation and a typed question, they return a probability distribution over the options in a single forward pass, without generating any text.

Three basic forms covering agent self-questioning scenarios:

Form	Returns	Use in the loop
Yes/No	Probability the statement holds	Safety gates, retry decisions, escalation to human
Single choice	Probability of each option	Routing, tool selection, triage
Scoring	Probability of each grade	Relevance, risk, confidence

No parsing, no format drift, and no waiting for reasoning tokens to finish generating.

Typed question means the answer to the question is constrained to a predefined discrete space, such as a boolean, a fixed set of options, or a rating scale, rather than arbitrary natural language text. This constraint lets the model output map directly to a probability distribution: computing a probability value for each candidate answer, summing to 1. The benefit of typed questions is that they turn "judgment" from a generation task into a classification/scoring task—the output space is small and closed, so a single forward pass yields probabilities for all candidates without autoregressive decoding. In practice, "typed" also means the question itself needs clear semantic boundaries—options must be mutually exclusive and exhaustive, otherwise the probability distribution loses interpretability. This is also the most easily underestimated engineering detail when deploying decision models: if the question is poorly designed, even a well-calibrated model cannot provide usable gating signals.

Calibration Is the Key

Accuracy alone cannot support a gate. A gate needs a threshold, and a threshold needs probabilities that "mean what they say."

If the model says 0.9 and it's right nine times out of ten, you can auto-pass everything above 0.9, escalate the rest to humans, and predict the error rate before going live. Conversely, if the model has good average accuracy but its 0.9 could mean anything from 0.6 to 0.99, then no threshold is safe, and you're back to humans reading outputs the old way.

This is why decision benchmarks evaluate calibration, not just hit rate. Yes/no questions use Brier score, multiple-choice questions evaluate probability mass on the correct option, and scale questions use expected score error. A model can absolutely top the accuracy leaderboard yet be unusable as a gate because of poor calibration.

Brier score is a classic metric for evaluating the calibration of probability predictions, defined as the mean squared error between predicted probabilities and actual outcomes (0 or 1), with lower values being better. For yes/no questions, if the model predicts 0.9 and the actual occurrence rate is 0.9, the Brier score approaches the theoretical lower bound; if the model frequently gives extreme probabilities yet frequently errs, the Brier score rises significantly. Unlike accuracy, Brier score penalizes both "wrong direction" and "overconfidence," thus revealing whether a model "means what it says." Multiple-choice scenarios commonly use probability mass on the correct option, while scale questions use expected score error—the deviation between the expected value of the predicted distribution and the true grade. The common logic behind these metrics is: the value of a decision model isn't in guessing correctly most often, but in its output numbers being usable as true probabilities for setting thresholds.

What Changes in Practice

Once checks become cheap, you can afford the checks that previously didn't pencil out:

• Gate every tool call, not just the ones that look dangerous
• Score the relevance of every retrieved passage, rather than blindly trusting top-k
• Ask "did this step succeed?" after every step, rather than discovering failure at the end
• Keep a confidence number for every decision, so escalation rules are based on thresholds rather than guesses

The architectural change is actually small: the decision model sits inside the loop as a function call, rather than beside it as another agent.

How to Read Public Leaderboards

Public leaderboards for decision models now exist for reference, such as S1MB, covering 197 models and 137 benchmarks, ranked using rank aggregation rather than score averaging.

When comparing entries, keep three things in mind: which column is being cited, the date the data was read, and how many benchmarks the model was actually scored on. Most public leaderboards mix self-reported and verified numbers.

Open-weight options already exist in this category, meaning you can test on your own traffic before deciding whether to purchase. A reasonable evaluation approach: run your own decision logs through candidate models and see whether the probabilities hold up at your target threshold.

Borda score is a rank-based aggregation method: rank models on each benchmark, then assign points by rank position (e.g., first place gets N points, second gets N-1), and finally sum across benchmarks. Unlike score averaging, Borda aggregation is not affected by differences in the scale of scores across benchmarks, nor does it require all models to participate in all benchmarks. But it has two properties to watch: first, models that didn't participate in a benchmark typically don't get scored on it, so models participating in fewer benchmarks may rank artificially high due to a "smaller denominator"; second, rank aggregation masks absolute gaps—when two models have close Borda scores, the actual capability difference may not be small. Therefore, when citing a leaderboard, in addition to noting the column name and read date, you should also confirm the number of benchmarks the model was actually scored on, to avoid misreading "participated in few but ranked high" as comprehensive leadership.

A Current Reference Point

If you want a concrete model to test this approach, the S1MB English v1 leaderboard shows Darwin-27B-ZTC-v9 ranking first among 197 models across 137 benchmarks with a Borda score of 90.38 (read on October 20, 2026). The model uses the Apache-2.0 license, so you can run your own decision logs through it directly rather than trusting the leaderboard.

hf download FINAL-Bench/Darwin-27B-ZTC-v9 systemone_config.json --local-dir .
transformers serve FINAL-Bench/Darwin-27B-ZTC-v9 --decision-config systemone_config.json

The same leaderboard also publishes a task-average column, where this model ranks second at 72.20. When citing, please specify which column you're referring to.

One-line Summary

Decision models replace the high-frequency textual generative self-checks in agents with a single forward pass that returns a probability distribution, and calibration is the key to their usability.

Paragraph Summary

The article points out that much of the token consumption during agent operation doesn't come from task execution, but from high-frequency self-questioning checks like safety gates, relevance judgments, and format validation. Under current architectures, every judgment requires full text generation, which is costly, forcing teams to disable checks. Decision models offer another form: given a situation and a typed question, a single forward pass returns a probability distribution over options, with no text generation or parsing. The core challenge is calibration, not accuracy—only when probability meanings are reliable can thresholds be used for auto-passing and escalation. Once checks become cheap, you can gate every tool call, score every retrieved passage, and validate success at each step. The article concludes by introducing the S1MB leaderboard and Darwin-27B-ZTC-v9 as a testable reference point.