Hands-on: Jev vs Three Cheap Models — the Cost Advantage Isn't That Big

A "No-Nonsense" Benchmark

Most of the numbers circulating about TypeSafe's Jev decision model come from the vendor itself. To get independent data, one developer ran a head-to-head comparison on the same gateway pitting Jev 1.13 against Mistral Small 3.2, Gemini 2.5 Flash Lite, and GPT-5 nano: eight fixed test cases, five rounds per model, for a total of 40 requests per model.

The conclusion is less dramatic than the rumors suggest: Jev is indeed cheaper and faster, but the multiples are nowhere near the head-turning "100x." More interestingly, the author found that two of his initial conclusions didn't hold up, and he publicly issued two corrections. This process of "questioning the vendor first, then questioning yourself" is more valuable as a reference than a single benchmark score.

Cost and Latency: The Real Gap Is 1.4x to 1.7x

Let's start with the hard data. On the cost dimension, Jev 1.13 averaged $0.0000188 per call, Mistral Small 3.2 came in at 1.4x, and Gemini 2.5 Flash Lite at 1.7x. In other words, against equally cheap chat models, Jev's cost advantage is only in the single-digit multiple range — not the order-of-magnitude gap you hear in marketing pitches.

On latency, Jev's median response time was 352 ms, Mistral 1,343 ms, Gemini 877 ms, and GPT-5 nano a whopping 7,504 ms. GPT-5 nano cost 18.3x more than Jev, with the vast majority consumed by reasoning tokens. Only when comparing against reasoning models and frontier models do Jev's multiples become dramatic — in one test case, GPT-5.6 Sol cost 32.5x more than Jev, and Opus 5 reached 184.9x.

This points to an easily overlooked selection logic: Jev's value-for-money advantage mainly shows up in scenarios where "you were already planning to use a reasoning model," not where "you were already using a cheap chat model."

Accuracy: Jev Doesn't Win

On 27 support tickets with clear correct labels, each model ran three times. Jev 1.13 and Mistral Small 3.2 both scored 27/27, while Gemini 2.5 Flash Lite and GPT-5 nano scored 26/27. In other words, the claim that "Jev is more accurate" finds no support in this round of testing.

What truly sets Jev apart from chat models is that it outputs a confidence value. On clear tickets the confidence was 0.979, dropping to 0.841 on six deliberately ambiguous tickets. The direction is right, but the gap is smaller than what threshold-based routing would require — if you plan to use confidence for traffic distribution, this level of separation may not be enough.

Confidence Calibration and Threshold Routing

Confidence is a probability estimate a classification or decision model assigns to its own judgment, typically ranging from 0 to 1. It doesn't improve accuracy by itself; its value lies in enabling "confidence-based routing": high-confidence requests are handled automatically, while low-confidence requests are escalated to a human or a stronger model.

Whether this mechanism works depends on whether confidence is "calibrated" — that is, when the model says 0.9, is the actual accuracy really close to 90%? If the model gives 0.98 on clear samples but still gives 0.90 on ambiguous ones, the separation is insufficient, and threshold routing degenerates into random routing, adding system complexity for nothing.

The article observes confidence of 0.979 on clear tickets and 0.841 on ambiguous tickets — the right direction but too small a gap, a classic signal of under-calibration. When evaluating confidence, common tools include reliability diagrams and expected calibration error (ECE), rather than just looking at averages. For teams planning to use confidence for traffic distribution, the recommendation is to first plot this curve on your own data before deciding where to set the threshold.

Two Corrections: Structured Output and Temperature Were the Culprits

The author's two initial conclusions were both later overturned by himself, and this is the most worthwhile part of the entire article.

Correction 1: "Type unreliability" was a test configuration issue. The initial version claimed chat models couldn't reliably return declared types — for example, Gemini returned true when asked to output a probability, and GPT-5 nano returned a string. The truth is that structured outputs weren't enabled. With just one change — adding response_format: { type: "json_schema", strict: true } — Gemini went from 2/5 to 5/5, and GPT-5 nano from 4/5 to 5/5. Type safety is just one line of configuration in the request body; it's not a reason to choose Jev.

Correction 2: Run-to-run drift came from sampling. At default temperature, chat models gave different answers on identical inputs (Gemini: 26, 25, 26; GPT-5 nano: 24, 26, 26). After setting temperature to 0, all models scored identically across three runs. The so-called "instability" was actually sampling randomness, not a model capability flaw.

After both corrections, only latency and cost still hold. On the schema-enabled support ticket use case, Jev took 386 ms, Gemini 896 ms, Mistral 1,672 ms, and GPT-5 nano 7,006 ms.

What This Test Shows and What It Doesn't

The author is quite restrained about the limitations: one machine, one afternoon, one gateway, five runs per cell; the 27 tickets were written by the author himself, covering only a single task and limited to English. In his words, this is enough to refute a generalized conclusion, but not enough to establish one.

For technology decision-makers, the value of this test isn't in giving an answer to "should I use Jev" — it's in demonstrating a method: reproduce vendor numbers on your own gateway and tasks, incorporate request parameters (structured outputs, temperature) into variable control, and be willing to publish corrections when results go against your initial judgment.

The full tables, per-case latency, vendor original numbers, and several independent reproductions are all at jev-agent.com/benchmarks; the reproduction scripts can be run via python3 -m bench.headtohead --repeats 5 and python3 -m bench.accuracy. The author also discloses that he operates jev-agent.com and has no affiliation with TypeSafe AI.

Background

Structured Outputs and JSON Schema Strict Mode

Structured Outputs is a capability offered by mainstream LLM APIs today: developers can attach a JSON Schema to the request, requiring the model's returned JSON to conform to the schema's fields, types, and required constraints. OpenAI's response_format: { type: "json_schema", strict: true } is a concrete implementation of this mechanism — strict: true means the server enforces syntax-level validation on the model output, rejecting or retrying when the schema isn't satisfied, rather than merely "suggesting" in the prompt that the model comply.

This is fundamentally different from the traditional "write in the prompt that it should return JSON" approach: the latter relies on the model's goodwill, and type drift, extra fields, and boolean/string confusion are all common; the former pushes constraints down to the decoding or post-processing layer, greatly improving reliability. The correction in the article holds precisely because the initial test only used prompt constraints without enabling schema strict mode, leading to a missing configuration being misdiagnosed as a model capability flaw.

Note that strict mode usually requires the schema itself to be simple enough (avoiding overly deep nesting and complex union types), and support varies considerably across vendors — when comparing across models, you must verify one by one whether it's actually in effect.

Temperature Parameter and Sampling Randomness

When generating each token, a large model outputs a probability distribution over the vocabulary, then samples from it according to some strategy. temperature controls the "steepness" of this distribution: the higher the temperature, the greater the chance of low-probability tokens being selected, and the more divergent the output; as temperature approaches 0, sampling degenerates to approximate greedy decoding, selecting the highest-probability token each time, so results are highly reproducible on identical inputs.

Therefore, getting different answers from the same input across multiple runs at default temperature (commonly 1.0 or 0.7) is normal behavior of the sampling mechanism, not model "instability" or "capability degradation." The article's finding that all three runs scored identically after setting temperature to 0 is direct verification of this mechanism.

There are two practical implications: first, for benchmarking or scenarios requiring reproducible results, you should explicitly set temperature: 0 (some vendors also offer a seed parameter to further fix the random source); second, user-facing conversational products actually need to retain some temperature, otherwise responses will seem rigid and repetitive. Confusing these two scenarios makes it easy to misread a design choice as a quality flaw.