Ollama Decision Models Tested: Probability-Based Judgments in 90 Milliseconds

Ollama's latest release introduces a new class of model — Decision Models. Instead of generating text, they pick from a set of candidate answers you provide and return a full probability distribution. The headline numbers from the official announcement: an average of 90 milliseconds per decision, ~76% accuracy on public benchmarks, and zero cost since everything runs locally.

These models follow the Type-C interface specification, designed specifically for high-frequency, low-unit-value classification tasks. This article draws on a breakdown of the official blog post and benchmark data to map out the model's capabilities, how to use it, and the constraints you need to understand before shipping to production.

Why a Dedicated Decision Model?

Using a general-purpose LLM for a simple yes/no judgment sounds lightweight — but it isn't cheap. Remote API calls cost money and carry network round-trip latency. Yet these kinds of judgments tend to be exactly the high-frequency, high-volume, low-per-call-value workloads.

There's another problem: general-purpose models "think before they answer." A simple classification can burn hundreds of tokens, and the resulting prose is useless to downstream logic — which only needs a label and a probability. Free-form text also can't give you a reproducible probability distribution that a program can act on directly.

Decision models turn this into a local HTTP request: put the text to evaluate in State, put your questions in Questions, and the local model answers everything in a single request — no reasoning steps, just direct scoring over candidate answer tokens. On Nimble 9B running on an M5 Max, that averages 90ms with only 1–4 output tokens. This is the payoff of "pattern matching."

The official README puts it plainly: "There's no reasoning step" — and that's the fundamental reason it's fast. The model reads the prompt once, then directly scores the answer tokens rather than generating a full explanation before extracting a conclusion. Output tokens are just 1–4; input tokens range from 155 to 841.

The "pattern matching" framing comes from information retrieval. Here it means: instead of autoregressive next-token generation, the model treats each candidate answer as a known token sequence and directly computes the conditional log-probability for each, then normalizes into a probability distribution. This is fundamentally different from how a standard language model reasons about "what comes next" — that requires a full generation chain; this only requires a single forward pass for scoring. That's why output tokens are 1–4 rather than hundreds: the model emits the highest-scoring answer token, not a chain of reasoning. The engineering trade-off is flexibility: the model cannot respond with anything outside the candidate set, but in exchange you get deterministic outputs and a quantifiable probability distribution — both critical for programmatic consumption.

Four Question Types and Request Structure

Decision models support four question types, and you can mix them freely in a single request:

• Choice: Pick one from 2–26 options. Returns the most likely option and the probability for each.
• Null (boolean): Returns a single probability of truth between 0 and 1.
• Score: Positions a value on an ordered scale of 2–26 levels. Returns a probability-weighted average score.

Only two fields are required in the request. State is the text to evaluate — it can be a string, object, or array (the latter two are serialized as JSON text, not treated as conversation messages or multimodal input). Questions is 1–64 named questions, each consisting of a Type, Instructions, and usually Criteria.

One key point: answers are not passed to subsequent questions — each question is scored independently. The request body is capped at 64KB, and Nimble's context window is 8,192 tokens.

The Score type has four rules that are easy to get wrong. First, criteria is an array, not an object — you write 2–26 level descriptions ordered from lowest to highest, and the order itself defines the scale. Second, levels are zero-indexed, not one-indexed. Third, score is a probability-weighted average (sum of each level's probability multiplied by its index), not the index of the highest-probability level. Fourth, the response always returns four fields — score, legend, probabilities, and confidence — all of them.

The setup cost is minimal: upgrade to version 0.3.5 or higher, pull the Nimble model, and send a POST request to the /decision endpoint. Local requests don't require an API key; even if the official TypeSafe SDK requires one, Ollama ignores it.

Confidence: What It Actually Measures (and What It Doesn't)

The official Pokémon maze demo is intuitive: a 187-step maze where, at step 31, the model judges that going right is correct with 0.95 probability — one request per step. But the more instructive moment is when the two sides show 0.65 and 0.35 — Confidence is just 0.07. That's the model telling you it genuinely isn't sure.

So the right use of Confidence is to set a threshold: below it, route to a human or trigger a fallback — don't just hardcode taking the max.

There's a common misreading worth correcting: high Confidence does not mean the answer is correct. It measures how concentrated the probability distribution is. The official docs explicitly state that "higher values do not guarantee correct answers." A 0.9 probability doesn't mean 90% accuracy on your data — you have to measure the right threshold on your own dataset.

Technically, Confidence here is closer to the inverse of distribution entropy: when the top option's probability is far higher than all others, the distribution is concentrated and Confidence is high; when probabilities are near-uniform across options, the model has no strong preference and Confidence approaches zero. For two options at 50/50, Confidence is 0, not 0.5. This is distinct from calibration in machine learning — calibration asks whether the model's output probabilities match actual observed frequencies (i.e., do predictions at 0.9 probability actually hit 90% of the time?). Confidence only reflects how "certain" this particular prediction is. An uncalibrated model can give extremely high Confidence on a wrong answer, which is exactly why the official docs warn it doesn't guarantee correctness. In practice, Confidence is best used to prioritize human review — not as a direct proxy for accuracy.

Tool Call Auditing: A Safety Gate Before Agents Execute

One of the most practical applications of decision models is auditing tool calls. Take an rm -rf command to delete a directory: the model returns a harm probability of 0.996 in 138ms. This is the safety gate before an agent executes a tool — exceed the threshold and the action gets blocked, queued for human approval, or routed through an approval workflow.

On choosing between Choice and Null: use Choice for "which one is it," use Null for "is it or isn't it." The closer the question type matches the semantics of your question, the more reliable the probabilities. Null is simpler — it only returns a single "probability of being true" field. Since true and false are complements, there's no probabilities array and no confidence.

In a real-world ticket triage scenario, all three types can be mixed in one request: team assignment uses Choice (returns "billing" with 0.985 probability, Confidence 0.92), refund eligibility uses Null (probability 0.997), and urgency uses Score (0.815 on a 0–2 scale). That request used 841 input tokens and just 4 output tokens. Three answers, three consumption patterns: assignment routes via a switch statement, refund above threshold auto-triggers a workflow, urgency sets the SLA tier.

Benchmark Results: Nearly on Par with Jeff

Bespoke Labs published this benchmark covering 33 public datasets and 3,880 human-annotated decisions. Of the four models tested, Jeff 1.1B leads at 76%, Nimble 9B is nearly tied at 75.7%, Jeff 1.4B reaches 73.3%, and the smallest 0.8B variant scores only 63.5%.

Breaking it down by type is more revealing: Jeff leads on content moderation by nearly 11 percentage points, while Nimble edges ahead on safety guardrails by 0.8 points. The overall average gap is under 1.5 points (74.8 vs. 76.0). Nimble pulls ahead on Score tasks (54.6 vs. 50.1); Jeff leads clearly on boolean tasks.

Note that the low absolute value on Score (54.6) is a methodology artifact — it requires exact highest-level matching to count as correct, so any probability shift counts as wrong, not a signal that the model is unusable. One additional data point: on Bespoke Labs' outflow-level evaluation, Nimble scores 90.12, while its base model Qwen 3.5 9B scores only 66.36 — fine-tuning genuinely elevated its discrimination capability.

The Jeff model series is Bespoke Labs' in-house decision-specialized family, ranging from 0.8B to 1.4B parameters, forming the other major camp alongside Ollama's Nimble series (fine-tuned from a Qwen base). The 33 public datasets in the benchmark cover typical decision scenarios including content moderation, safety guardrails, and intent classification; the 3,880 samples are all human-annotated to avoid label noise from fully automated evaluation. Notably, Nimble 9B has roughly 6x the parameters of the largest Jeff variant, so their near-identical accuracy suggests Jeff has a significant edge in parameter efficiency. Nimble's advantage is that as a native Ollama model, the call chain is shorter and deployment is simpler. When choosing between them, the trade-off between parameter efficiency and operational convenience is more meaningful than raw accuracy numbers alone.

Vision Capabilities and Six Production Constraints

Decision models can also process images. Clave and Clave Flash require version 0.3.5.1 or higher and must include vision weights. The rules: images must be read from file and encoded as base64 yourself (URLs and data URLs are not accepted); all questions in a request share the same image set; state is still required and provides decision context — the image tells the model "what it sees," while state tells it "what this screenshot is about." A typical use case is automatically routing UI screenshots attached to support tickets to the right team.

Six constraints to know before going to production:

1. It can only choose from the candidates you provide — it won't write explanations, emit nested JSON, or quote source text.
2. Questions are scored independently — if two conclusions need to be consistent, you have to enforce that in your own code.
3. The prompt must fit in 8,192 tokens. Shorter prompts yield better accuracy.
4. A 0.9 probability does not equal 90% accuracy.
5. Add a fallback option for cases where no candidate is a good match.
6. Decision models are not yet in Ollama's official Python/JS libraries.

Practical recommendations: delegate explanatory copy to downstream template rendering; add consistency assertions in code; short State plus short Instructions is the highest-leverage accuracy improvement; measure your threshold on your own data before setting a cutoff; use Confidence to prioritize human review of the most ambiguous cases; consider wrapping a thin SDK layer that you can swap out once official support lands.

Summary

Local execution, zero cost, reproducible outputs — that's the real value proposition of decision models. Boolean tasks are reliable enough for moderation and blocking use cases. Score tasks are better suited to coarse bucketing with human review in the loop. This isn't a replacement for general-purpose models. It's a purpose-built tool for "high-frequency, low-value judgments" — a category that general models have always handled inefficiently — handing that work to a pattern-matching specialist.