The Hidden Flaw in OpenAI's Decisions API: Reorder the Options, and the Probability Shifts by 13 Points

A developer ran a thousand experiments on OpenAI's new Decisions API. First, a weighted coin biased 70% toward heads — the API performed nearly perfectly, sampling heads at almost exactly that rate.

Then he switched to a jar of marbles: five red, three blue, two white. The true probability of drawing red was 50%. Yet the API chose red 86% of the time.

He reordered the answer options, moving "red" from first to last. That number dropped to 73%.

Same question. Same jar. Same model. The only difference was the order of the options.

This is the API that's had people buzzing for an entire week. Announced at DevDay, powered under the hood by a dedicated gpt-6-luna model, it's aimed squarely at the rapid-decision market that TypeSafe's Jev just ignited. And amid all the noise, this quiet experiment — where probabilities drift just because you rearranged a list — went almost completely unnoticed.

If you're planning to hand your routing logic to a model, you should probably know about this marble jar first.

What the Decisions API Actually Is

The idea is straightforward — refreshingly so, in a way you wish more launches would be. Rather than generating text token by token, the model returns one of three typed answers:

• predicates: yes or no, with a probability
• choices: pick one from a fixed list you define
• scores: rate something against a rubric

No paragraphs, no filler, just a fast judgment call.

OpenAI claims up to 10x speedups by skipping the long-form generation step. In theory, each call takes around 150 milliseconds rather than the 1.6 seconds text generation requires. For high-concurrency routing, that math is appealing.

The tradeoff: the model can't explain itself or show its reasoning. What you get is a verdict and a number.

This kind of "structured output" design isn't something OpenAI invented. The community has been using constrained decoding techniques — going back to Function Calling and JSON Mode — to force models into outputting only token sequences that conform to a specific schema. Constrained decoding works by masking the probabilities of all illegal tokens at inference time, forcing output into a predefined vocabulary space. Much of the Decisions API's speed advantage comes from this: when the output space shrinks from the entire vocabulary down to "yes/no" or a short list, the model doesn't need to unfold a full reasoning chain step by step, KV cache reuse rates go up, and time-to-first-token drops dramatically. The cost is equally clear: Chain-of-Thought research has repeatedly shown that letting a model "say" its reasoning steps first significantly improves final answer accuracy. Trading that away for speed means voluntarily giving up part of your accuracy ceiling.

Head-to-Head with Jev

There's one question everyone asks about this launch: is it a copy?

TypeSafe's Jev went from a viral novelty to a $7.5 billion valuation and over $100 million in annualized revenue in what felt like weeks. It defined the category.

OpenAI followed almost immediately with a direct competitor. The sentiment on r/singularity was pretty well summarized: "This cuts into a big chunk of the market TypeSafe was targeting." The top-voted reply was more dismissive:

Wait, isn't this just copying Jev — essentially an ultra-lightweight Luna?

The benchmark numbers land somewhere between the two. LangWatch ran both on "LLM-as-judge" tasks:

	OpenAI Decisions	TypeSafe Jev
Accuracy	75.2%	63.7%
Median Latency	243 ms	429 ms
Cost per 1K decisions	$1.09	$0.42

OpenAI is faster and more accurate; Jev costs less than half as much. The data crowd called it "a statistical tie with a price gap."

The "Slow" Nobody Tweeted About

The more interesting complaints came a few days later. A thread on r/LLMDevs titled "Jev alternative" described the Decisions API as "disappointingly slow" — not exactly what you'd expect from a launch that promised 10x speed.

Same old story. Benchmarks are a toy; real traffic isn't.

Someone tried using it to route customer support tickets and found the median latency acceptable, but the tail was maddening — those few extra seconds were enough to make the system feel broken.

I think the deeper issue is OpenAI's frame of reference. Compared to its own generation endpoints, this is genuinely fast. Compared to a small model doing the same classification task, that gap closes quickly.

One more thing the release notes left out: both models fall apart when actual math is involved. Summing up invoice line items? A hard no for either. These models are making judgments, not calculations.

The "long-tail latency" problem has a precise vocabulary in distributed inference: P50, P95, P99 latency. A system with a P50 of 243 milliseconds can easily have a P99 above 2 seconds — meaning 1 in every 100 requests will feel noticeably slow to the user. For something like customer support ticket routing, that 1% tends to cluster at peak load, exactly when inference servers have the longest queues. OpenAI's shared API endpoints show significant queuing delays during traffic spikes, which behaves completely differently from a self-hosted small model. Production workloads that need reliably low latency typically require purchasing Provisioned Throughput to bypass the shared queue — and that changes the cost math entirely.

Microsoft Enters Three Days Later

Then the strangest thing happened. Three days after OpenAI's public beta, Microsoft launched its own decision-scoring model: Microsoft-Decision-1.

The twist: it was built by post-training Alibaba's open-source Qwen3.5-9B. Not an OpenAI model. Microsoft — OpenAI's biggest partner — built its own fast judgment layer on top of a Chinese open-source model.

It's also cheaper: $0.042 per million input tokens, no charge for output, plain text, 32k context.

Microsoft claims it outperforms across 36 benchmarks covering nearly 150,000 blind-test questions, running 35x faster than flagship models like GPT-6 Sol. They also plan to migrate it onto their own MAI model.

I read this as a statement: Microsoft doesn't want to depend on OpenAI for the cheap, boring, high-concurrency judgment layer.

So now there are three players competing for the same job: making fast, cheap calls on a single question. Whoever wins becomes the default brain inside every routing system.

Qwen3.5 is an open-source model from Alibaba's Tongyi Qianwen series, released under the Apache 2.0 license, which allows commercial use and post-training modifications. Microsoft's choice to build on it via supervised fine-tuning (SFT) and reinforcement learning alignment — rather than using OpenAI's models — has several commercial motivations: inference costs on open-source base models are far lower than API fees; it avoids a single-vendor dependency for core infrastructure; and it's a differentiation move for the Azure AI platform, which sells OpenAI services but also needs its own moat. This aligns with strategies Google and Meta have pursued in recent years: on standardized, commoditized inference tasks, fine-tuned open-source models can now compete directly with closed-source flagships — and the "judgment layer" is precisely the kind of task easiest to standardize.

Stepping Back from the Benchmarks

I have a folder on my desktop called "launch week." Every time a company announces "the fastest model ever," I screenshot it and drop it in. There are forty files in there now. Not one of them is still the fastest model ever.

gpt-6-luna, Jev, Decision-1, Dots. Every name was the future for exactly one week.

I mentioned this folder to a friend and she said the real pattern is that companies are running out of nouns. I think about that every time a new launch drops.

This is what the category looks like now: something new arrives, and within a month it's a commodity row in a spreadsheet full of tradeoffs.

And the naming. Your infrastructure now depends on models with names that sound like indie bands. Jev sounds like a fantasy novel character. Luna sounds like a perfume. Decision-1 sounds like a Star Trek robot.

The Honest Part

Most people don't actually need this. Worth saying clearly.

If you're routing a few thousand decisions a day, a regular model with a JSON schema does the same job. Slower, yes — but free, because you're already paying for that model anyway.

This category only makes sense at scale: millions of content moderation calls, enterprise-grade ticket routing, those agent loops that used to burn thousands of tokens just to choose between two options.

And if you do use it, don't trust the raw probabilities. The marble experiment is right there. A model whose confidence score drifts 13 points just because you reordered a list is giving you a feeling, not a measurement.

Treat the scores as signals, build your own thresholds, and test the order of your answer options before you ship.

One more honest note: the 10x speedup only holds if you were generating text you didn't actually need. If your workflow requires writing a paragraph, this API is useless to you. It's a switch, not a writer.

I keep coming back to the marble jar. The coin was perfect; the jar was a mess. That experiment tells the whole story of this API: it's great when the question is clean, and it gets weird when reality shows up with noise, ordering effects, and bias baked in.

I'd use it for triage. I wouldn't ask it for truth.

Maybe the real lesson is simpler: the hype cycle moves faster than calibration. Everyone rushed to debate "who killed Jev," and almost nobody stopped to ask — when you shuffle the answer order, do these probabilities still hold up? That question is worth more than all the hot takes from launch day combined. And I still haven't found a good answer.