Training a Jev Decision Model on a Consumer GPU: A System 1 Practice with 19x Lower Latency

In AI Agent architectures, up to 80% of decision nodes actually only need "System 1"-style intuitive reflexes, rather than the lengthy token-by-token autoregressive generation of language models. Microsoft's Microsoft-Decision-1 and TypeSafe AI's Jev have already commercialized this approach, but the cost and latency of frequent cloud API calls are not acceptable to every team. Based on a complete set of on-device benchmark records, this article breaks down how to train a decision model with the same Decision Head architecture on a consumer GPU, and provides reproducible latency and calibration data.

The Essential Difference Between Decision Models and Traditional LLMs

In the past, the standard way to classify tickets or logs was to write a long prompt instructing the model to "return only JSON," then hard-parse it on the application side with Pydantic or regex. For a binary question, the model would autoregressively emit dozens to hundreds of tokens behind the scenes, often taking several seconds, and occasionally spit out a malformed string that crashed the system.

Decision models take the opposite route. A traditional language model's final layer projects the hidden state into a vocabulary space of about 152,064 dimensions, which is extremely expensive to compute. Jev, Cloudflare's Clef, and Unsloth's integrated Decision Head instead pool the final hidden state of the sequence, attach a compact Decision Head, and run one more forward pass, computing the probabilities of a few hundred dimensions of options all at once. The inference process does not need a decoding loop at all.

Microsoft's Microsoft-Decision-1 claims to make decisions 35x faster than GPT-6, and its base model is actually Alibaba's open-source Qwen3.5-9B—an open-source foundation plus classification training, sold directly. This also means you can fully replicate the same architecture locally.

Pooling in the Decision Head architecture refers to compressing the sequence hidden states output by the Transformer's final layer (shape [sequence length, hidden dimension]) into a fixed-dimension vector so that a lightweight classification head can be attached. Common approaches include taking the hidden state of the last non-padding token (last-token pooling), averaging over all tokens (mean pooling), or taking the maximum (max pooling). This step strips away the sequence-length dimension, so the subsequent Decision Head only needs to process one vector, avoiding repeated computation for each position—and it is the key to a decision model completing classification in a single forward pass.

QLoRA and LoRA: LoRA (Low-Rank Adaptation) freezes the original model weights and injects low-rank decomposition matrices into modules such as attention layers for fine-tuning, greatly reducing the number of trainable parameters. QLoRA further quantizes the base model weights to 4-bit (e.g., NF4 format) and dequantizes them to BF16 for computation during the forward pass, reducing the training VRAM requirement for 4B-class models from tens of GB to under 8 GB. In this article, rank=16 and alpha=16 are the LoRA rank and scaling coefficient: rank determines the capacity of the newly added parameters, and alpha controls the magnitude of their updates.

The Complete Training Workflow on 24GB VRAM

The benchmark used a machine with 24GB VRAM, chose unsloth/Qwen3.5-4B as the base model, paired it with 4-bit quantization and LoRA (rank=16, alpha=16), and used the open-source decision benchmark LocalLLaMA/typed-decisions as the dataset, containing 1,120 training samples, 80 validation samples, and 400 independent test samples.

Metric	Measured Result
Base model and quantization	Qwen3.5-4B (4-bit QLoRA)
Peak training VRAM	7.95 GB
Training time	910.6 seconds (about 15 minutes)
Zero-shot initial accuracy	30.5%
Validation accuracy after training	80.5%
Accuracy on 400 unseen test samples	74.9%
Expected Calibration Error (ECE)	0.187 → 0.036
Average latency per forward pass	191.4 ms

VRAM overhead is quite restrained: the 4B model peaks at only 7.95 GB for the entire training run in 4-bit. Extrapolating at the same ratio, switching to the Qwen3.5-9B base model used by Microsoft would put VRAM at roughly 14 to 16 GB, so if resources allow, you can go straight to the 9B model.

An easily overlooked highlight is the calibration performance: ECE dropped from 0.187 to 0.036, meaning the model's output confidence is no longer just decoration but can serve as a threshold criterion in production environments.

A/B Benchmark: The Gap Between 3.7 Seconds and 0.19 Seconds

To verify "just how much faster it is," a strict A/B test was run on the same GPU: on one side, a traditional prompt asked Qwen3.5-4B to emit JSON; on the other, the fine-tuned Decision Head was used.

Comparison Item	Traditional Generative Prompt	Decision Model (Decision Head)
Average inference latency	3,679.8 ms	191.4 ms
Computation mechanism	Token-by-token generation (~64 tokens consumed)	Single forward pass
Format reliability	Frequently emits <think> reasoning chains, unfinished	100% guaranteed JSON-compliant format
Confidence information	None	Built-in calibrated probability distribution vector

When testing the traditional generative model, the new-generation reasoning model even automatically emits <think> tags, hits the max_tokens limit halfway through writing, and fails to output even the closing JSON brace. The decision model delivers a clean probability dictionary in 191 ms. In scenarios with thousands to tens of thousands of calls per day, this gap magnifies into a highly significant difference in cost and experience.

Multi-Task Expansion and Stress Testing with Abnormal Inputs

Real-world situations are usually not so simple. Below are the measured behaviors in several edge cases.

Marginal Cost of Multi-Question Expansion: Only 16 ms per Additional Question

Answering multiple questions about the same piece of text simultaneously (e.g., determining both the responsible team and the urgency level) is a common requirement. Measured latency breakdown: asking one question averages 175.28 ms, asking two questions simultaneously averages 191.35 ms—the marginal cost of one additional question is only 16.07 ms. The reason is that the backbone feature encoding of the entire context only needs to be computed once; the Decision Heads attached afterward are just lightweight linear projections. Even with 5 classification judgments mounted at once (team, urgency, sentiment, PII leakage, language), inference latency increases by at most 60 to 80 ms.

Division of Labor Between Backbone and Decision Head: In the Decision Head architecture, the base model (Backbone) is responsible for encoding the input text into a semantic vector, and its computational cost depends on the input length and model size. The Decision Head is one or more small linear projections attached after the pooled vector, mapping the hidden dimension to the number of classes. In multi-task scenarios, all tasks share the same Backbone encoding result, and each task mounts an independent Decision Head. Therefore, the cost of adding a judgment task comes only from that one lightweight forward pass and is almost independent of input length. This is the architectural root of the "16 ms marginal cost" and explains why multiple classifiers such as team, urgency, sentiment, PII leakage, and language can be mounted simultaneously without significantly increasing latency.

Probabilistic Response to Irrelevant Junk Messages and Ambiguity

When completely irrelevant text is fed in (e.g., the lyrical line "Autumn wind passes through the golden leaves under the moonlight"), the model's highest confidence plunges from the normal 99.9% to 54.3%, with probability spread across the options (billing 15.7%, technical_support 30.0%, account_management 54.3%). This property can be used directly with a simple threshold to filter abnormal inputs, preventing the online service from being poked and prodded at will.

Expected Calibration Error (ECE) measures the deviation between the model's output confidence and its actual accuracy: predictions are bucketed by confidence, and the weighted average of the difference between the average confidence and the true accuracy in each bucket is computed. The lower the ECE, the more it means that when the model says "90% sure," it is indeed about 90% correct. After ECE dropped from 0.187 to 0.036, the model's output probability distribution became production-usable—a threshold can be set (e.g., route to a human or mark as abnormal when confidence is below 60%) without misjudgments caused by overconfidence or underconfidence. This is also one of the core advantages of decision models over generative LLMs: generative models cannot provide calibrated class probabilities.

Cross-Domain Scenario Validation

Scenario 1: Routing High-Controversy Customer Service Tickets

Given the ticket "credit card was charged twice and a same-day refund is requested," the model assigns 99.97% confidence to the billing category and rates the duplicate-charge issue as 89.31% high priority. Maintaining high confidence on both types of judgment simultaneously is exactly the advantage of the Decision Head multi-task architecture.

Scenario 2: NGS Quality Anomaly Alert in a Bioinformatics Pipeline

In a genetic testing analysis pipeline, quality checks on sequencing batches often require engineers to manually inspect the data. Given a description of a FastQC report showing an average Phred score of Q18, a 42% duplication rate, GC content abnormally surging to 65%, and a 28% drop in DeepVariant call rate, the model neither simply lets it pass (pass probability only 0.63%) nor arbitrarily demands a rerun; instead, it requires human review with 77.21% confidence and judges the risk level as moderate with 84.7% confidence.

Scenarios that involve patient information are exactly where the value of a local decision model shines: data never leaves the premises, and it comes with interpretable probabilistic confidence, which better meets compliance requirements than sending everything to a cloud LLM.

Specialized Models Are the Cost-Effective Choice for Enterprise AI

Over the past two years, the industry has become accustomed to relying on large language models to solve complex logical reasoning and expression problems. But back in real system architectures, the thousands to tens of thousands of intent routing, security filtering, or state judgments per day only need fast, accurate, resource-efficient conditioned reflexes. If a system needs to throughput a large number of decisions every day, bearing the token costs and hundreds of milliseconds of network latency of cloud APIs is not cost-effective.

A single consumer-grade graphics card can complete Decision Head training and deployment locally. For scenarios that need to handle sensitive data, a decision model with built-in probabilistic confidence and no data leaving the domain is the architecture choice that truly offers high cost-effectiveness for enterprise AI applications. Rather than continuing to pay cloud inference fees for every simple binary judgment, it is better to spend 15 minutes training your own System 1.

One-line Summary

Train a Decision Head decision model on a consumer GPU in 15 minutes, reducing latency from 3.7 seconds to 0.19 seconds.

Paragraph Summary

The article contrasts the essential difference between traditional LLM token-by-token generation and the Decision Head architecture: the latter pools the sequence hidden state and attaches a lightweight classification head, so a single forward pass outputs the probabilities of all options, completely skipping the decoding loop. The author fine-tuned Qwen3.5-4B with 4-bit QLoRA on a 24GB VRAM machine; peak training VRAM was only 7.95GB, training took about 15 minutes, test-set accuracy was 74.9%, and ECE dropped from 0.187 to 0.036. The A/B benchmark showed latency falling from 3679.8ms to 191.4ms, with 100% format reliability. Multi-task expansion adds only about 16ms per additional judgment; when faced with irrelevant input, the highest confidence automatically drops to 54.3%, which can serve as an anomaly detection signal. The article also validates the value of local decision models in compliance and interpretability through two scenarios: customer service ticket routing and NGS quality alerts.