Stop Parsing LLM Essays: How the Verdict Layer Ends the Regex Era

Every agent you've ever deployed hides an awkward secret in its control flow: somewhere in a corner, a regex is parsing the model's English prose back into a boolean. This week, the industry finally shipped a primitive that can kill that hack.

OpenAI's Decisions API entered public beta — POST /v1/decisions, returning only typed answers: a probability, a choice, or a score. Days later, Microsoft released Decision-1 in Foundry: a post-trained Qwen3.5-9B designed specifically for single-shot decision scoring, priced at just $0.042 per million input tokens with free output. The same week, SGLang added the /v1/decisions endpoint to its nightly build, and vLLM merged /v1/systemone. The interface consensus has formed: stop generating decisions, start scoring them.

The Duct-Tape Solution We've Been Using

Take an honest look at your current stack. It looks like this:

response = llm.chat("Classify this lead as HOT, WARM, or JUNK. Reply with JSON only.")
label = json.loads(response.text)["label"]   # pray

You pay output tokens for reasoning, then throw them away. You wait through a full generation just to get a yes/no answer. Every few hundred calls, the model gets "creative" and your parser swallows an exception at 2 a.m.

JSON mode and structured outputs have reduced the crashes, but the architecture is still wrong: you're using a text generator as a classifier because there was no servable classifier at the time. Now there is.

JSON mode and structured outputs refer to the constrained decoding capabilities offered by model providers (such as OpenAI's JSON Mode, JSON Schema, and grammar-constrained guided decoding). They restrict the token distribution during sampling so that output necessarily conforms to a preset format, reducing parse failure rates from "one crash every few hundred calls" to near zero. But their scope is limited to format legality, not semantic correctness: the model can still fill a legal JSON with a label you never defined, or give an answer unrelated to the input. More critically, constrained decoding only solves "parsing," not "probability." What you get through JSON is still a discrete label with no calibrated confidence, which cannot support threshold policies. This is exactly the point of the article: structured outputs only make the duct tape stronger, whereas the verdict layer replaces the entire duct-tape approach with typed, probability-bearing primitives.

The Verdict Layer Pattern

The alternative consists of three parts.

1. Typed Decision Calls

Fixed-option inputs, calibrated probability outputs. OpenAI's three forms cover virtually every control flow: predicate (probability a condition holds — your if statement), choice (an option plus probability — your switch), score (position on an ordered scale — your priority queue).

2. Threshold Policies, Not Gut Feelings

This is the part most tutorials skip. Without a policy wrapped around it, the number is useless:

score = decide("Is this support ticket urgent?", ticket.text)
# → {"urgent": 0.91, "normal": 0.09}
if score["urgent"] >= 0.85:
    page_oncall(ticket)
elif score["urgent"] >= 0.60:
    human_review_queue(ticket, score)   # score travels with the ticket
else:
    auto_resolve(ticket)

Pass the score along with the ticket. When a human reviews it, what they see isn't a paragraph of argumentation — it's a traceable number.

3. Audit Logs

Every decision lands in a log carrying the input hash, score, threshold outcome, and timestamp. When a decision goes wrong, you don't argue with a paragraph of prose — you look directly at the number, the threshold, and the input. This is also why the pattern is compliance-friendly: OpenAI offers zero data retention support and HIPAA compliance eligibility in the beta, because a scored, logged decision has an auditability that prose can never match.

What Vendors Are Claiming (A Critical Read)

OpenAI Decisions API: public beta, runs only on gpt-6-luna, $0.10 per million input tokens, output and cache free. The official line is "about 10x faster than the Responses API" — but this is measured against OpenAI's own API, with no disclosed benchmark and no head-to-head accuracy comparison against competitors. Images must be passed inline as base64; hosted URLs and file IDs are both rejected.

Microsoft Decision-1: announced by Achint Srivastava, VP in the Office of the CTO, available in Foundry and coming soon to OpenRouter. A post-trained Qwen3.5-9B, with plans to later rebuild the base on Microsoft AI and OpenAI models. $0.042 per million input tokens, output free. Claims: highest accuracy on 36 benchmarks (roughly 150,000 held-out questions), 4.5x faster than Quyet-1.0-Large, 35x faster than GPT-6 Sol at P50, a decision flip rate of only 1.3% under eight input perturbations, and 5,250 safety requests completed across 11 benchmarks. All vendor self-reported. Trust, but verify with your own eval set.

One price comparison is interesting: TypeSafe's Jev — which an independent review called "OpenAI's direct answer product" — lists the same $0.042 per million input tokens. The category now has a price anchor, and it's cheap.

Perturbation robustness is a core metric for decision models. So-called input perturbations include synonym rewrites, punctuation and casing changes, insertion of irrelevant sentences, minor typos, and even swapping the position of key conditions. A classifier that judges only by surface cues is highly prone to flipping its conclusion under perturbation, which in a production agent means the same ticket could take a completely different path because of an extra comma. Microsoft's claim of a 1.3% decision flip rate under eight perturbations matters because it measures stability rather than absolute accuracy — a model with high accuracy but a high flip rate makes threshold policies meaningless. For verification, it's advisable to construct your own perturbation set and focus on flipped samples rather than only overall accuracy.

Migration Checklist

1. Grep for duct-tape code. Find every prompt ending in "reply with JSON only" or "respond with just the label." That's your migration list.

2. Convert one classifier first. Pick the highest-frequency, lowest-risk one (triage, tagging, routing). Shadow-run the typed endpoint alongside your parser for a week and compare disagreement rates.

3. Write down your threshold policy. Calibrated or not, probabilities need documented cutoffs and escalation bands. "Block below 0.85, human review from 0.6–0.85" is a policy; a gut-feel cutoff is a bug.

4. Log every verdict. Input hash, score, outcome, timestamp. This is the cheapest reliability upgrade in the entire pattern.

5. Use it to score your agents. Microsoft explicitly targets rubric-based scoring of AI responses and agent actions. Your agent output gets scored by a cheaper model before it reaches the user — that's the containerized pattern productized.

Shadow mode is a migration technique in which the old and new systems run in parallel, logging only without taking effect: the typed endpoint is called as usual, but its results are not used for real actions. Instead, they're written to logs alongside the existing parser's results to compare disagreement rates, latency, and cost. Its value lies in turning the "model accuracy" question into a quantifiable delta metric: when the two agree, switching is risk-free; when they disagree, you can directly inspect the specific samples to judge whether the new scoring or the old parsing is more correct. A common switch criterion is a disagreement rate below some threshold, combined with manual review confirming that the new system is no worse than the old on the disagreeing samples. Shadow runs are typically scheduled for one to two weeks, enough to cover weekday and weekend traffic distributions as well as low-frequency but high-risk edge inputs.

Why This Goes Beyond the API Itself

The author, Shivam Kumar, runs a decision engine called Jev in production at VisionQuantech, used nightly for lead scoring, reply classification, and build-priority ranking. His agent stack has always had a decision stage — THINK, ACT, then DECIDE: a small scoring model renders a verdict on every critical step, logs it, and thresholds it. It has always been the quiet stage, and it has always been the stage that keeps the system honest.

This month, the industry formalized it as a servable primitive, with two vendors and two open-source stacks converging on the same interface in the same week. When that happens, the interface itself is the story, not the leaderboard.

Build the verdict layer. Your 2 a.m. self will thank you.

Background Notes

Calibration refers to the property that a model's output probabilities match true frequencies: when the model says "0.9," roughly 90% of such samples are indeed positive. Confidence given by LLMs through prompts or generative classification tends to be overconfident, and using uncalibrated probabilities directly to set thresholds produces a large number of misjudgments. Common corrections are temperature scaling or isotonic regression on a validation set, mapping raw scores to empirical frequencies before writing documented cutoffs. The article emphasizes that "calibrated or not, probabilities need documented cutoffs and escalation bands," where the escalation band refers to the gray zone between automatic handling and outright blocking, for example 0.6–0.85 triggering human review. The width of the gray zone is essentially a trade-off between precision and cost: the wider the band, the fewer misses but the more human effort required; the narrower the band, the higher the automation rate but the higher the risk.

One-line Summary

OpenAI, Microsoft, and major inference stacks converged on a typed decision interface in the same week, marking the industry's shift from generating decisions to scoring them.

Paragraph Summary

The article notes that agent control flows have long relied on a "duct-tape solution" of regex or JSON parsing to convert model prose back into booleans — wasteful of output tokens and fragile. This week, OpenAI's Decisions API, Microsoft's Decision-1, and support for the /v1/decisions endpoint by SGLang and vLLM formed an interface consensus around "scoring rather than generating" in the same week. The author proposes the verdict layer pattern: typed decision calls (predicate, choice, score), documented threshold policies, and audit logs carrying input hashes, scores, and timestamps. Vendor-claimed performance and pricing are all self-reported, and $0.042 per million input tokens has become the category's price anchor. Migration should start with high-frequency, low-risk classifiers, shadow-running before switching.