Microsoft Decision-1: A Specialized Small Model for Scoring AI Agent Decisions

Microsoft has released a small language model called Microsoft-Decision-1 that doesn't generate text or perform complex reasoning. It does one thing: return calibrated probability scores for each option in a given set of choices. This seemingly narrow capability targets a long-overlooked pain point in AI agent engineering — the speed and cost of every individual decision step.

What Microsoft Released in Foundry

According to Microsoft's official announcement, Decision-1 is a model built for a narrow task. Achint Srivastava, CVP at Microsoft's Office of the CTO, described its positioning this way: "Unlike large language models designed to generate text or reason through complex problems, decision models are purpose-built to deliver structured outputs that software can act on immediately." The model is now live on Microsoft Foundry, with access planned through OpenRouter.

Its operating mode defines its use case. Given a fixed set of candidate answers, Decision-1 returns a calibrated probability score for each option — meaning the application layer controls the threshold, while the model handles the scoring. Whether routing customer service tickets, checking tool call compliance against a rubric, or scoring AI-generated responses, it all reduces to the same call: one scenario, one closed list of options, and one number per option.

This is a much smaller job than a chat model, and Microsoft has priced it accordingly. As documented by Windows Report, the rate is $0.042 per million input tokens, with output tokens free. That pricing changes the cost arithmetic for many scenarios: a customer service system scoring 200,000 inbound messages per month would pay under $9 in input token costs — turning checks that many teams had dropped for cost reasons into routine per-message operations.

Calibrated Probability is the key feature that distinguishes Decision-1 from a standard classification model. A "calibrated" model means its output confidence scores are statistically aligned with actual accuracy: if the model assigns 80% confidence to a batch of predictions, roughly 80% of those predictions should actually be correct. Many large language models suffer from overconfidence — outputting 90% confidence when actual accuracy is only 70%. A poorly calibrated model forces engineering teams to empirically tune thresholds through trial and error, while a well-calibrated model allows scores to be used directly as triggers for business logic — for example, "below 70%, escalate to a human." This is why Microsoft listed calibration as one of five major engineering challenges in its announcement — it's not a nice-to-have, but a prerequisite for a specialized scoring model to be actually usable in production.

How Scoring Models Work Alongside Generative Models

A pattern that has become common in production-grade agents is placing a cheap "judge" alongside an expensive "writer." The large model handles drafting; the small model answers a bounded question — what to do next.

Microsoft offered a latency case in its announcement that makes the rationale clear without needing benchmarks: "Adding 100 milliseconds to each of 20 sequential decisions, for example, adds two seconds to a workflow." In customer service or financial workflows, agents make exactly this many interdependent decisions, each inheriting the latency of the underlying model.

There's also a cost logic Microsoft didn't spell out explicitly: when a general-purpose model is asked to score a fixed list, you still pay for all the reasoning text it produces along the way — and that's exactly what a specialized scoring model eliminates. The engineering value of a decision model lies in stripping text generation from a step where nobody reads the output.

Five Engineering Challenges Microsoft Identified

The announcement lists the challenges in the order teams actually encounter them — the list reads like a technical spec sheet for other builders: speed, generalizable quality, robustness, calibration, safety.

Generalizable Quality

Quality across unseen tasks is the hardest of the five to simply "buy." Microsoft described its validation approach: dozens of benchmarks held out blind during training, covering routing, ranking, long-context, multilingual, and out-of-distribution tasks, plus a set of top public models from the JevBench leaderboard compared across 36 additional benchmarks. Microsoft reports achieving the highest accuracy in this 36-benchmark comparison covering nearly 150,000 questions, and claims P50 latency approximately 35x faster than GPT-6 Sol.

Robustness and Calibration

Robustness comes with a reproducible number. The announcement states: "Microsoft-Decision-1 changes its decision on an average of 1.3% of perturbations, with zero flips when option descriptions are reworded or options are reversed or shuffled." For anyone setting escalation thresholds, that flip rate is the critical figure. Microsoft framed the calibration goal in operational terms: "Applications use confidence to decide when to act, defer, or request human review, so a 90% prediction should be right about nine times out of ten on a representative sample."

Internal Test Results Microsoft Published

Vendor benchmarks describe a model on a leaderboard; internal deployments describe a model in a workflow — a different kind of evidence. Microsoft published four such cases.

According to the announcement, the Xbox Research team used Decision-1 to process over 10,000 open-ended responses and comments from surveys, Steam, and Twitter/X, categorizing them into a fixed set of topics defined by researchers. The team found the model "comparable in quality to GPT-6 Sol while running more than 14x faster and at 200x lower cost." The Copilot team, measuring chat and agent response quality, found it "comparable to GPT5.6 Luna and 100x faster." On-call engineers used it for knowledge retrieval during live incidents. Microsoft Discovery used it to score experiments in an adaptive replanning loop, reporting that model scoring was "46x more consistent than LLM-based scoring and 3x faster."

Looking at these four cases together reveals a pattern worth noting: every internal case had a fixed set of labels that existed before the model did — researcher-selected topics, quality rubrics, relevance judgments. The model handles classification; the meaning of the categories is determined by someone else.

Designing a Test Plan for Your Own Tickets

Microsoft didn't publish a deployment guide; evaluation is left to the buyer. Fortunately, the plan is short enough to run within a week.

Pick a narrow task with real cost consequences: triaging a support queue, categorizing customer feedback, or running pass/fail judgments on AI drafts against a rubric. Build an annotated set of a few hundred of your own cases — vendor benchmarks describe their edge cases, and your tickets carry your own. Run the new model on those cases and compare accuracy, latency, and cost against your current approach on the same dataset.

Then read errors by direction, not average. A wrong "proceed" and a wrong "escalate" are two differently priced events, and thresholds set on overall accuracy will mask that asymmetry. Rewrite the same inputs and observe how often decisions flip — the vendor's 1.3% figure is a target to reproduce, not a fact to assume. Until your own data is in, keep humans reviewing consequential actions, and log every score alongside the outcome it predicted, because that log becomes the calibration data the next decision depends on.

Pricing and the Rebase Path

Two release details affect planning. First, the pricing — $0.042 per million input tokens, with outputs free — sets a manageable cost for a check running on every message or every tool call. Second, the base model: Microsoft states it was post-trained on Qwen3.5-9B for this task, and will soon rebase the model onto other systems, including models from Microsoft AI and OpenAI.

Rebasing is essentially behavior change wearing a version number. When the underlying model changes, scores that previously calibrated to a given threshold may shift. So a team that tracks thresholds should also track which model version those thresholds belong to. This habit costs nothing and protects the thing a decision model must preserve: a score meaning the same thing across versions.

Microsoft has placed a cheap, calibrated score at the point where an agent chooses its next action, and published its own test numbers alongside pricing that makes high-frequency checks affordable. What remains for the buyer is small and concrete: your cases, your baseline, your thresholds, and the model version recorded next to them — that's the entire difference between a scoring model on a spec sheet and one running in production.

Background

Post-training refers to the process of narrowing a pre-trained foundation model's behavior toward a specific task through supervised fine-tuning (SFT), reinforcement learning (RLHF/RLAIF), or other alignment techniques. Microsoft's choice of Qwen3.5-9B as a starting point means Decision-1 inherits that base model's language understanding while being specifically optimized to output only structured probability distributions rather than free-form text. Rebasing means migrating the same post-training pipeline onto a new foundation model to gain improved underlying capabilities or alignment with a newer version. This process will inevitably cause the model's numerical outputs on identical inputs to drift, which is why tracking the correspondence between model version numbers and thresholds is not optional engineering hygiene — it's the minimum requirement for maintaining predictable behavior in a decision system.