Behind Jev Router's Top Finish: 117 of 118 Calls Pointed to GPT-6.1 Sol

A routing model took first place on a benchmark, but 117 of its 118 calls went to the same model—a model that only ranked third when it competed on its own. This isn't magic; it's a sober story about benchmarks, routing mechanics, and statistical noise.

Event Recap: First Place and Third Place Are the Same Thing

TypeSafe's Jev Router (typesafe/jev-router on OpenRouter) topped the Forge 2.0 agentic coding benchmark, scoring 0.6398 and 0.8523 across two builds for a median of 0.746, ahead of Claude Opus 5.5's 0.6745. But OpenRouter's generation records show that 117 of 118 calls went to GPT-6.1 Sol, with only 1 going to GPT-6 Astra. And GPT-6.1 Sol's own two-build median was just 0.5933, placing third.

In other words, the champion got its win by routing calls to the third-place model. That sounds like a loophole, but more precisely, it exposes the part of the "routing" layer that is nearly impossible to measure in benchmarks.

GPT-6.1 Sol and GPT-6 Astra: These two models, which recur throughout the article, are key to understanding the whole event. Sol and Astra are different variants within the same GPT-6 generation, with different positioning—Sol leans toward general-purpose agents and long-context tasks, while Astra may differ in specific capabilities or pricing. Jev Router chose Sol for almost all of its 118 calls; the single call that switched to Astra produced a one-off bill of $1.80, accounting for 44.87% of Build 1's total cost. This shows the router's choices aren't random but based on some decision logic—yet when tasks are highly homogeneous, that logic degenerates into "always pick the same one," making the router's performance essentially identical to that of the selected model itself.

The Nature of a Router: Writes Nothing, Only Makes Choices

Jev is not a large language model. According to OpenRouter's documentation, Jev is TypeSafe's System One decision model, which only returns typed answers with probabilities and produces no text, reasoning, or explanation. Jev Router uses it to select a model and reasoning effort for each request; the selected model generates the response, and billing is at that model's rate.

This aligns with the academic definition of a router. LMSYS's RouteLLM paper describes it as a strategy that dynamically switches between a strong model and a weak model to optimize the cost-quality tradeoff; RouterBench is premised on the idea that "no single model can optimally handle all tasks"; and RouterEval even found that a good router can "outperform the best single model in the pool." The problem is that when the pool contains only a few models and tasks are highly homogeneous, a router easily degenerates into "always picking the same one."

RouteLLM, RouterBench, and RouterEval: These three are representative works in the routing research field and form the academic backdrop for understanding Jev Router. RouteLLM, proposed by LMSYS, centers on dynamically switching between a strong model and a weak model to balance cost and quality; RouterBench is a standardized benchmark premised on the idea that "no single model can optimally handle all tasks"; and RouterEval further found that a good router can "outperform the best single model in the pool." Together these studies define the router's value proposition: achieving cost optimization or quality gains through intelligent selection. But their experimental setups typically include multiple models with clearly differing capabilities and diverse task types, whereas Jev Router's performance on Forge 2.0 lacks exactly these two conditions—a small pool and homogeneous tasks—causing the router to degenerate into "always picking the same one." This is precisely the gap between the academic definition and engineering reality.

What the Data Reveals: 117/118 Means the Router Was Barely Routing

The two Jev builds together had 120 generation IDs (118 routed calls plus 2 auxiliary Gemini 2.5 Flash calls from goose itself). Querying OpenRouter's /api/v1/generation endpoint for each revealed:

• Build 1: 74 Sol, 1 Astra, 1 Gemini Flash, score 0.6398, cost $4.02
• Build 2: 43 Sol, 1 Gemini Flash, score 0.8523, cost $1.15

The single Astra call accounted for 44.87% of Build 1's total bill. It carried a 142,694-token prompt with zero cache hits, costing $1.80 for a single call, while the 74 Sol calls in the same build totaled just $2.22. This confirms the cost trap OpenRouter itself has warned about: switching models means rebuilding the input cache, and in long-context agentic tasks a single switch can swallow a large chunk of the budget.

Note that this is only performance on "our task." OpenRouter's model page shows that across all traffic, DeepSeek^Ad V4.1 Flash accounts for about 36% of tokens and Sol about 18%. That the router almost exclusively picks Sol in long-context Forge builds doesn't mean it would do the same under mixed traffic.

The Structural Problem with the Benchmark: Two Builds Can't Distinguish the Router from the Model

Forge 2.0 requires each model to run two independent builds and ranks by median. But the median gap between two builds of a single model is already 0.0832; Jev's own two builds differ by 0.2125, and Gemini 4 Argon in the same group differed by as much as 0.4983 between its two builds (0.7926 vs. 0.2943). Under that much noise, a 0.1527 median lead isn't solid.

More critically, consider the comparison groups. Jev Router and Gemini 4 Argon both belong to comparison group 1, while Opus and Sol belong to group 2. The leaderboard itself states: "compare models within the same comparison group." Comparing Jev and Sol across groups isn't fair by the authors' own rules. Jev's builds ran on goose 3.0.116, while Sol's ran on 3.0.110—73 commits apart. Although the toolset and system prompt for the first request were the same, the engine difference is itself a variable.

Moreover, Jev's low-scoring build of 0.6398 differs from Sol's high-scoring build of 0.6495 by only 0.0097. Under the rule of the authors' own comparison tool forge2_compare.py—which declares a winner only when all of one side's builds exceed all of the other's—this pairing should be judged "unresolved."

Comparison groups and median ranking: Forge 2.0 uses a mechanism of "two independent builds per model, ranked by median" and divides models into different comparison groups, with the leaderboard explicitly requiring comparisons "within the same comparison group." Jev Router and Gemini 4 Argon both belong to comparison group 1, while Claude Opus 5.5 and GPT-6.1 Sol belong to group 2. Comparing Jev and Sol across groups isn't fair by the leaderboard's own rules. More critically, the median gap between two builds of a single model is already 0.0832; Jev's own two builds differ by 0.2125, and Gemini 4 Argon in the same group differed by as much as 0.4983 between its two builds. At that noise level, a 0.1527 median lead isn't solid—it's smaller than the gap between Jev's own two builds, meaning the ranking conclusion can't be statistically supported.

An Unverifiable Hypothesis: The Router May Have Raised Reasoning Effort

Goose sent reasoning: {effort: "medium"} in every build, but OpenRouter's documentation explicitly states that with Jev Router this is only a suggestion, not a guarantee. The 45 fields in the generation records contain no effort information, but native_tokens_reasoning offers a clue: when calling Sol directly, reasoning tokens never exceeded 516 across 95 calls; after routing, 19 of the 74 Sol calls in Build 1 exceeded 516, peaking at 3,327.

This is consistent with the hypothesis that the router raised reasoning effort, but it could also be other factors not yet ruled out. The author suggests: if you use a router in your own agent, be sure to send the X-OpenRouter-Metadata: enabled request header, which will list candidate models along with their effort and routing rationale in the response.

Reasoning effort and native_tokens_reasoning: Reasoning effort is a parameter introduced by model providers such as OpenAI that controls the token budget a model spends on internal reasoning before generating a response, typically divided into tiers like low, medium, and high. Goose sent reasoning: {effort: "medium"} in every build, but OpenRouter's documentation explicitly states that with Jev Router this is only a suggestion, not a guarantee—the router may override the setting. The 45 fields in the generation records contain no effort information, but the native_tokens_reasoning field offers a clue: when calling Sol directly, reasoning tokens never exceeded 516 across 95 calls; after routing, 19 of the 74 Sol calls in Build 1 exceeded 516, peaking at 3,327. This is consistent with the hypothesis that the router raised reasoning effort, but it could also be other factors not yet ruled out. The author suggests sending the X-OpenRouter-Metadata: enabled request header, which will list candidate models along with their effort and routing rationale in the response, thereby verifying this hypothesis.

Check by Check: Which Three Checks Did Jev Win?

Forge 2.0's score is "test score" multiplied by a "reliability coefficient." The reliability coefficient penalizes the worst performance in each of 19 test groups: if the worst test completely fails, that group is multiplied by 0.90. Jev's two builds had test scores of 0.9409 and 0.9856, versus Sol's 0.9498 and 0.9383—essentially tied at the average check level. The gap comes from reliability: Jev at 0.68 and 0.8647, Sol at 0.6838 and 0.5724.

There are three checks that Jev won in both builds and Sol won in neither: posting a sprint comment as a user in Atlassian documentation format, an event pipeline for tracking issue re-estimation, and an in-app AI explanation button returning an answer. Two of these Sol only partially lost points on (0.8333 and 0.5), not complete failures. There is no check that Sol won in both builds while Jev won in both.

Jev's own two builds disagreed on 12 checks, while Sol's two disagreed on 22. Build 1 hit 47 storage-limit errors on one seed site, while Build 2 had none; Build 1 rejected 1 of 958 user-facing quota requests, while Build 2 rejected 0 of 984. Each check takes the lowest score among three seed sites, so one bad site can drag down an entire build.

Cost Comparison: Routing Has No Extra Fee, but It Carries Switching Risk

Claude Opus 5.5's two builds billed $42.50 for a median of 0.6745; Jev Router spent $5.17 to get 0.746; Sol on its own spent $2.38 to get 0.5933. Jev's per-call cost ($0.0536 and $0.0268) was higher than calling Sol directly ($0.0245 and $0.0256), mainly due to that one Astra switch.

Routing itself is free, but it introduces switching risk. After removing the Astra call, Build 1's 74 Sol calls cost $0.0299 each, still slightly higher than calling directly. In other words, on this task, the cost advantage of routing doesn't exist.

How to Verify the Router's Real Behavior Yourself

The author provides actionable verification steps: keep the generation id for each call; query the model and router fields with GET https://openrouter.ai/api/v1/generation?id=<id>; tally the model distribution—if one model handles nearly all calls, compare the router's results against that model's direct-call results within the same test harness, same version, and same week; and send X-OpenRouter-Metadata: enabled to see reasoning effort and routing rationale.

The judgment criterion is also straightforward: when the router's lead is smaller than the gap between its own two builds, the ranking isn't trustworthy. Jev leads Sol by a median of 0.1527, while its own two builds differ by 0.2125.

Conclusion: Do You Want the Routing, or the Model?

If the goal is GPT-6.1 Sol's performance on this kind of long agentic task, calling Sol directly gets you the same model and avoids the switching risk, at the cost of giving up the reasoning-effort adjustment the router may (or may not) bring. If the goal is cost optimization under mixed traffic—which is exactly what routers were designed for—this benchmark doesn't test it at all.

TypeSafe itself wrote in an article about benchmarks: "Run your own private evaluations, be skeptical of public results, and don't amplify every number or chart you see." That advice applies to this leaderboard too.

The real next step is clear: on the same goose version, run third and fourth builds for both Jev and Sol, with the metadata header enabled. Until then, the distance between first and third place may just be the shape of statistical noise.

One-line Summary

Jev Router topped the benchmark, but 117 of its 118 calls routed to the third-place model Sol, exposing a routing layer that is nearly impossible to measure in benchmarks.

Paragraph Summary

TypeSafe's Jev Router topped the Forge 2.0 agentic coding benchmark with a median of 0.746, ahead of Claude Opus 5.5's 0.6745. But OpenRouter's generation records show that 117 of 118 calls went to GPT-6.1 Sol, while Sol on its own only ranked third (median 0.5933). That means the champion got its win by routing calls to the third-place model. The article digs into the nature of a router—it only makes choices and produces no text—and its tendency to degenerate into "always picking the same one" when the pool is small and tasks are homogeneous. The data also reveals the cache-rebuild cost risk of switching models: the single switch to Astra accounted for 44.87% of Build 1's total bill. In addition, the benchmark itself has structural noise: Jev's own two builds differ by 0.2125, larger than its 0.1527 median lead over Sol, and the two belong to different comparison groups and ran on different goose versions—not a fair comparison by the leaderboard's own rules. The article closes with actionable verification steps and notes that the real next step is to run more builds on the same version with the metadata header enabled.