Microsoft Decision Model in Practice: How Probabilistic Output Is Revolutionizing AIOps Root Cause Localization

When the Decision Model Stops "Talking": Probabilistic Output Delivers Cost and Efficiency Gains for Operations

In AIOps (intelligent operations), log anomaly detection and microservice root cause localization have long been two core challenges. Traditional approaches either rely on manually defined rules or use large language models (LLMs) to generate analysis reports. Microsoft's Decision-1 model offers a completely new approach: it does not generate any natural language, only probabilistic values—noul for yes/no decisions, choice for multiple-choice decisions, and score for ordered scoring. Each call consumes only about 1 output token.

This "silent" design fits operations scenarios perfectly: engineers do not need the model to explain "why"; they only need a fast, precise probabilistic signal to support decision-making. Developer Pedro Sakuma open-sourced a complete reproduction project on GitHub, systematically comparing Decision-1, gpt-4.1-nano, reasoning models, and non-LLM baselines on two public AIOps datasets.

HDFS Log Anomaly Detection: The Optimization Path from 49% to 86%

The True Nature of the Dataset

The experiment uses the Loghub HDFS_v1 dataset, grouping log sequences by data block. Each data block should have a complete lifecycle: NameNode allocates the block, three DataNodes (replication factor of 3) receive and acknowledge it, PacketResponder terminates, and finally it is registered with the NameNode.

The key finding came from analyzing a calibration set of 300 blocks: 38 normal blocks contained anomaly keywords (such as "got exception while serving"), while 14 anomalous blocks had no error logs. The true signal of an anomaly is lifecycle completeness—whether the expected steps repeat three times and which stages are missing. A simple rule of "any anomaly event appears" achieved only 71% on the test set.

Replication Factor is the number of copies of a data block stored in a distributed storage system such as HDFS. HDFS defaults to a replication factor of 3, meaning each data block is written to three different DataNode nodes. This mechanism ensures data availability when a single node fails. The replication factor directly determines how many times events such as "allocate," "write," "acknowledge," and "terminate" should appear in a data block's lifecycle—when the replication factor is 3, these steps should repeat three times. Therefore, any insufficient or missing stage may indicate an abnormal block write. Understanding the replication factor is the prerequisite for interpreting the "lifecycle completeness" logic later in this article, and it is also the key domain knowledge for converting log text into structured validation rules.

The Failure of the First Attempt

The initial prompt was very intuitive: the state was event templates plus counts, and the question was "Is this block anomalous?" Decision-1 scored 49% (AUC 0.68), even worse than the 71% rule baseline. The saved outputs revealed the problem: probability correlated 0.43 with event count and only 0.14 with the label. Short blocks (≤10 events, all anomalous) averaged 0.45 anomaly probability, while normal blocks with 15-20 events received 0.66—the model overreacted to lexical cues and log volume, and misclassified "missing events" as healthy.

AUC (Area Under the ROC Curve) is a common metric for measuring the ranking ability of a binary classification model, ranging from 0.5 to 1. A value of 0.5 means the model's judgments are no better than random guessing, while 1.0 means perfect separation. AUC does not depend on a specific threshold, so it is suitable for evaluating the quality of a model's probabilistic signal, whereas accuracy is greatly affected by threshold choice. In this article, Decision-1's original score of 49% (AUC 0.68) means that in terms of accuracy it was even worse than the rule baseline, but the AUC of 0.68 shows that its probability ranking was still slightly better than random—the main problems were that the default threshold of 0.5 was inappropriate and that the construction of input features misled the model. This also explains why the article later emphasizes that "thresholds must be calibrated."

The Breakthrough of Structured Prompting

Two key adjustments completely changed the results: at the state level, lifecycle_summary was computed from HDFS domain knowledge (expected replica count, allocation count, write start/completion count, acknowledgment termination count, registration count, missing steps, error event count); at the question level, "Is it anomalous?" was changed to "Is the lifecycle complete?"

Final results on 1,200 test blocks: Decision-1 with calibration achieved 86% accuracy (threshold 0.15), gpt-4.1-nano achieved 79%, and gpt-6.1-sol (only 100 samples) achieved 66%. Decision-1's AUC reached 0.964.

Training a Classifier Is Still the First Choice

When labeled data and a stable schema are available, logistic regression achieved 95.1% with 50 labeled blocks and 99.7% with 8,000 blocks. Zero-shot scenarios are where decision models excel—86% usability without training and even when the schema changes frequently.

Zero-shot scenarios refer to a model performing inference directly on a target task without any fine-tuning or labeled examples. The related settings are few-shot (providing a small number of examples) and fine-tuning (updating parameters with labeled data). The advantage of a zero-shot decision model is that it requires no labeled data collection, no training pipeline, and can be quickly deployed when the schema changes frequently. In this article, when there are 50 to 8,000 labeled blocks, logistic regression reaches 95% or even over 99% accuracy under a stable schema, far exceeding the 86% of zero-shot LLMs; but when labels are scarce or the field structure changes often, zero-shot approaches such as Decision-1 become the only feasible option. This trade-off is the core basis for choosing a technical route.

Microservice Root Cause Localization: Practical Validation of 94% Accuracy

Task Design and State Construction

The RCAEval RE2-OB dataset injects 6 types of failures across 5 services into the 11 microservices of Online Boutique, for a total of 90 cases. The state received by the model is compressed by a script: metric changes 12 minutes before and after the failure (CPU, memory, sockets, disk, latency, error rate), trace span time changes, and log volume changes, retaining only the top items with the largest changes.

Service-Level Localization Performs Well

Decision-1 correctly identified the root cause service in 85 of 90 cases (94%), exceeding the "largest metric deviation" rule baseline (82%, McNemar p=0.013). Among the 17 cases where the rule and the model disagreed, 14 were correctly resolved by Decision-1—the rule was often misled by downstream symptoms (9 times incorrectly selecting frontend), especially in packet loss and latency failures.

The model also made predictable errors: 3 times it misidentified the caller checkout as the root cause of an email latency failure, and 2 times it was misled by downstream services in packet loss scenarios.

The McNemar test is a statistical test for paired binary data, used to determine whether two classifiers have significantly different error patterns on the same set of samples. It focuses on samples where the two models' predictions disagree: the number of cases where one is correct and the other is wrong. In this article, the rule baseline and Decision-1 disagreed on 17 cases, of which 14 were correct for Decision-1; the McNemar test gives p=0.013, meaning this difference is statistically significant (usually p<0.05 is considered significant), so Decision-1's advantage is unlikely to be caused by random fluctuation alone. Introducing this test avoids drawing excessive conclusions from a tiny difference in accuracy numbers.

Failure Type Identification Remains a Weakness

Failure type identification achieved only 51% accuracy. CPU and latency failures had obvious features (13 correct out of 15 each), but packet loss failures were 0 correct, and socket exhaustion was only 2 correct. No field in the state directly describes "packet loss" or "socket exhaustion," and all models (including gpt-6.1-sol at 54%) performed poorly here.

Practical recommendation: Use Decision-1 to rank the direction of investigation (narrowing 11 services down to 1, taking 240ms), rather than directly naming the failure type.

Cost and Latency Comparison

Model	Average Latency	Cost for 1,380 Calls
Decision-1	240ms	About $0.015
gpt-4.1-nano	728ms	About $0.05-0.07
gpt-6.1-sol	2052ms	About $0.56

At the same call volume, Decision-1 is about two orders of magnitude cheaper than reasoning models.

Practical Takeaways

• State engineering beats model selection: Let the model receive precomputed lifecycle summaries and metric changes, not raw counts
• Ask structured questions: "Is the lifecycle complete?" is better than "Is it anomalous?"
• Thresholds must be calibrated: 0.5 is a poor default and should be optimized on a holdout set
• Train a classifier when labels are available: Small models comprehensively outperform LLMs on stable schemas
• Decision models do not explain: Keep human review for major decisions

For the complete code and reproduction steps, see the GitHub project pedrosakuma/decision-1-aiops.

One-line Summary

Decision-1 replaces natural language explanations with probabilistic output, achieving high accuracy at extremely low cost in AIOps anomaly detection and root cause localization.

Paragraph Summary

Microsoft's Decision-1 model generates no text at all, only probabilistic values (noul/choice/score), consuming about 1 output token per call. Developer Pedro Sakuma open-sourced a reproduction project on GitHub and systematically evaluated it on two AIOps datasets: HDFS log anomaly detection and microservice root cause localization. In the HDFS task, by changing the state to a computed lifecycle summary and the question to "Is the lifecycle complete?", accuracy rose from 49% to 86%, with an AUC of 0.964. Microservice root cause localization reached 94% accuracy, exceeding the largest metric deviation rule baseline (82%). However, failure type identification was only 51%, with 0 correct for packet loss failures. In terms of cost, Decision-1 is about two orders of magnitude cheaper than reasoning models, with only 240ms latency. Conclusion: state engineering beats model selection, and logistic regression remains optimal when labeled data is available.