GPUStack Decision Routing in Practice: Using Jev to Let the Gateway Auto-Select Models

GPUStack v2.3.0rc2 previews an advanced routing capability worth hands-on testing — using the Jev decision service to select the appropriate generative model for each request. This article walks through the entire chain: understanding the role of the decision model, deploying two private models, integrating DeepSeek and Jev, and then verifying the gateway's actual routing choices across tasks of varying difficulty.

What the Decision Model Actually Does

The first step to understanding decision routing is distinguishing the responsibilities of two types of models. Generative models answer questions; decision models provide the basis for choosing which generative model to use.

A concrete example makes this clear: "Translate 'Good morning' into Chinese" and "Design a cross-region payment system" require completely different capabilities. The former is well within the reach of a small model; the latter may need a more powerful model to reason through fault handling and consistency issues.

Jev receives the request content, the candidate models, and their capability descriptions, then returns a structured selection rationale via the TypeSafe Choice interface. The GPUStack gateway uses this to select a model and forwards the request to the corresponding generative service — the final answer still comes from Qwen3 or DeepSeek^Ad.

This means the core of configuring decision routing is clearly describing which types of tasks each candidate model is suited for. The decision model does not establish business quality standards on behalf of administrators; the more closely capability descriptions match real workloads, the easier it is to validate routing outcomes.

The Problem Jev Solves at the Gateway Layer

GPUStack's existing model routing can place multiple targets behind a single endpoint. Fixed-weight routing answers the question "how much traffic should each model receive" — suitable for canary releases and traffic splitting. Jev decision routing goes further, answering "which model should handle this specific request."

For an assistant that handles both text and code, an ideal division of labor might look like this:

Task Type	Candidate in This Policy
Short translation, field extraction, factual Q&A, basic arithmetic	Private Qwen3-8B
Constrained algorithm implementation, SQL, localized code debugging	Private Qwen3-32B
Complex concurrent code, cross-region system design	External DeepSeek

Applications only need to use one route name; the division of labor is maintained by administrators on the GPUStack side. Adding or removing candidates and adjusting task boundaries does not affect how applications make calls. This is the gateway-level value of a decision model: centralizing task classification into routing policy so that different models can each do what they do best. Whether this actually improves quality or reduces costs still requires validation with real workloads — this article first validates whether task routing executes as configured.

The architectural difference between fixed-weight routing and decision routing is worth elaborating on. Fixed-weight routing (e.g., weighted round-robin) is a stateless traffic distribution mechanism: allocation ratios are determined before a request arrives, with no awareness of request content. This suits A/B testing or canary deployments, but cannot distinguish between "a simple translation" and "a complex system design" in terms of capability requirements. Decision routing is stateful and content-aware: the content of each request is sent to the decision service, which dynamically evaluates which candidate model is best suited for that request. The trade-off is an extra external call per request (to Jev), introducing additional latency. It is therefore appropriate for scenarios with clear requirements around quality or cost tiering — not every routing scenario warrants introducing a decision service.

Preparing Models and External Services

Start by preparing the three candidates responsible for generating answers, then integrate Jev for decision-making.

Deploy Qwen3-8B and Qwen3-32B-W8A8 via GPUStack. Both should show 1/1 available replicas; send one inference request each to confirm they respond normally.

Next, add DeepSeek under Providers: select the DeepSeek type, enter the API key, and test connectivity. The generative model used in this test is deepseek-flash.

Then add Jev: select TypeSafe Decision Service (Jev), set the endpoint to https://api.typesafe.ai, enter the TypeSafe API key, and set the decision model to jev-latest.

At this point, responsibilities are cleanly separated — Qwen3 and DeepSeek are routing targets responsible for generating answers; Jev is the decision service responsible for providing the basis the gateway uses to choose a target.

Configuring Decision Routing

Create a route named demo-smart and add the two privately deployed models and the DeepSeek model as regular route targets. Applications only need to request demo-smart.

In the basic settings, select Policy, enable Decision Service Routing, select the Jev provider routing-jev, and set the decision model to jev-latest.

For this test, set all three regular traffic weights to 0 and the decision contribution weight to 20, making it easier to observe Jev's effect. The decision contribution weight is a relative influence on scores, not a traffic percentage.

Capability Descriptions Must Precisely Match

Next, define the decision instructions and model evaluation criteria — together they answer two questions: what is the overall selection principle, and which tasks is each candidate suited for?

Setting	Core Content for This Test
Decision Instructions	Prefer the lowest-cost candidate that can complete the task: simple tasks go to 8B, moderate tasks to 32B, complex tasks to DeepSeek
8B Capability Description	Suited for short translation, extraction, factual Q&A, short summarization, basic arithmetic; context budget 8,192 tokens
32B Capability Description	Suited for SQL, algorithm implementation, localized debugging; context budget 16,384 tokens
DeepSeek Capability Description	Suited for complex reasoning, concurrent code with tests, distributed architecture design

The key names in capability descriptions must match the actual candidate model names: route-local-qwen3-8b, route-local-qwen3-32b, deepseek-flash. Do not use route aliases or provider names — otherwise they cannot be mapped to candidates. Additionally, the cost assumptions of 8B as low-tier, 32B as mid-tier, and DeepSeek as high-tier are part of the configuration; this test did not calculate hardware amortization or API costs.

One Route, Validating Three Model Selections

All tests in this section go through the same route demo-smart. The nine test inputs are nine different questions, not nine different routes — only the question content changes while the model field stays constant. We observe whether the gateway selects 8B, 32B, or DeepSeek according to the configured division of labor.

Record Expected Results First

Start with three representative inputs: a short translation expected to go to 8B, constrained interval-merging code expected to go to 32B, and cross-region payment system design expected to go to DeepSeek. Recording expectations beforehand avoids retrofitting explanations after seeing the answers.

Send Requests and Capture Headers and Responses

Use the API for formal testing to track which model was selected. Create a GPUStack inference key, save the request body as request.json with the model field set to demo-smart. When sending requests, save both response headers to response.headers and the answer to response.json.

Subsequent tests only change the question in messages, with max_tokens increased for code or longer responses — 1,024 tokens for interval merging, 1,536 tokens for system design.

Trace the Actual Model via Request ID

Read x-gpustack-request-id from the response headers, then use it to locate the corresponding Higress log entry. Check upstream_cluster to confirm the forwarding target and ai_log.model to confirm the model name. In this test, model-2-6.static corresponds to 8B, model-3-7.static to 32B, and provider-1.dns to DeepSeek.

One critical insight: the evidence is the combination of "one request ID + its response + the matching access log entry." An HTTP 200 only proves the request succeeded — it does not prove which model was selected.

Higress is Alibaba Cloud's open-source cloud-native API gateway built on Envoy, which GPUStack uses as the gateway layer for handling AI inference traffic. In this architecture, GPUStack manages model deployment and routing policy configuration, syncing policies to Higress. Actual request forwarding, plugin execution (including calling the Jev decision service), and load balancing all happen on the Higress side. The upstream_cluster field is the standard Envoy/Higress field for recording the actual forwarding target; its naming convention is generated by GPUStack when syncing configuration (e.g., model-2-6.static) and corresponds to the candidate sequence numbers shown in the routing UI. This is why log tracing is the only reliable way to verify routing decisions — the HTTP response itself carries no information about which upstream model generated it.

All Results Hit the Mark

In addition to the three representative inputs, factual Q&A, extraction, arithmetic, and SQL tasks were also tested. All nine inputs went through demo-smart:

Test Input	Expected Model	Actual Model
Capital of France	8B	8B
Short translation of "Good morning"	8B	8B
Extract city from text	8B	8B
Calculate 2+2	8B	8B
Constrained interval-merging code	32B	32B
PostgreSQL aggregate query	32B	32B
Analyze and fix async lock deadlock	32B	32B
Cross-region order payment system design	DeepSeek	DeepSeek
Async LRU cache with tests	DeepSeek	DeepSeek

All nine requests returned HTTP 200 with finish reason stop, and actual models matched expectations in every case. The translation, interval merging, and system design tasks were each repeated three times in non-streaming mode — model selection remained stable. Streaming versions also selected the same models. This demonstrates that the configured task division can be realized behind a single endpoint.

Correct Model Selection Does Not Equal Usable Answers

To assess whether this division of labor has concrete justification, we sent the translation, extraction, arithmetic, and interval-merging tasks to all three candidates simultaneously with the same inputs and budgets. The code task required correctly merging intervals, not modifying the input, and raising ValueError for invalid intervals. Generated code was executed in an isolated environment.

Acceptance Criterion	8B	32B	DeepSeek
Short translation, city extraction, 2+2	All correct	All correct	All correct
Merge result, empty input, touching endpoints	Pass	Pass	Pass
Does not modify original input	Fail	Pass	Pass
Raises ValueError for invalid intervals	Fail	Pass	Pass

8B modified the original input and missed the invalid interval check; 32B and DeepSeek both satisfied the constraints. This provides concrete evidence for "routing constrained algorithm implementation to 32B," but does not establish a universal ranking of model capabilities. Notably, while DeepSeek's generated async cache code passed all three included tests, the logs still contained unhandled Future exceptions — outputs from stronger models also require independent validation.

What Happens When Jev Fails

Model selection depends on Jev returning valid decision rationale. If Jev is unreachable, times out, or returns an invalid result, the decision plugin publishes no ranking opinion, and the gateway continues with the rest of the selection and forwarding process. This is fail-open behavior.

In an isolated failure-test route, we configured an unreachable Jev endpoint while keeping all three candidates healthy. The logs showed publishing no rank, DeepSeek happened to respond, and the final result was HTTP 200. This shows that processing can continue after a decision failure, but it does not mean failure will always route to DeepSeek.

Another input of approximately 79 KB did not succeed: Jev returned 400 with no ranking opinion, the request proceeded to 32B, which then returned 400 due to insufficient context budget. When there is no decision ranking, the gateway cannot guarantee selecting a model that meets the task's requirements. Additionally, a translation prompt attempting to instruct the system to ignore the policy and force selection of DeepSeek still went to 8B, but the answer did not meet requirements — again illustrating that model selection and answer quality must be evaluated independently.

Back to the Gateway: Walking the Complete Call Path

After configuration and validation, the division of responsibilities between GPUStack and the gateway becomes clearer. GPUStack manages candidate models, providers, and routing policies, and syncs configuration to Higress. When a request reaches the gateway, the plugin prepares candidate information; the decision plugin calls Jev for selection rationale; after the gateway selects the upstream target, it forwards to the private model or DeepSeek.

The Jev call and the generative call are two distinct paths: Jev participates in model selection, while Qwen3 or DeepSeek generates the answer. If the decision call does not produce valid rationale, the request may continue, but there is no longer any guarantee that the intended model will be selected.

How the Plugin Uses Jev Results

Internally, the plugin prepares candidates and task descriptions, calls Jev, converts the returned model probabilities into candidate scores, and then hands them off to the load-balancing finisher, which aggregates scores and selects the upstream target. This process is based on decision plugin source code commit 790256e. Note that the probabilities and confidence values in the diagrams are for illustrating the calculation process only, not actual measured responses. Confidence adjusts the influence of the decision opinion, not the probability that the answer is correct.

For applications, the entry point is always demo-smart. For administrators, the work is maintaining the capability boundaries of candidate models and continuously validating selections with representative tasks. This is the value of decision routing: placing the decision of "which model should handle this" inside the gateway, making multi-model specialization a configurable, testable policy.

This validation covered a total of 36 requests (9 different inputs through the same route, 12 same-task baselines, 6 repetitions, 3 streaming requests, 6 failure or policy-control cases). Routing and transport each met their targets; answer quality was evaluated separately. This article focuses on capabilities introduced in v2.3.0rc2, tested in a development environment, without sustained concurrency testing or cost-benefit analysis.

TypeSafe Choice (the interface implemented by Jev) is a structured decision interface designed so that the output of AI decisions can be consumed directly by programs, rather than generating free text. Compared to directly asking an LLM to output "choose model A," a structured interface can return a probability distribution or confidence score for each candidate, making it easier for downstream systems to perform weighted aggregation. GPUStack's decision plugin merges these scores with the base weights of candidate models, and the load-balancing finisher then determines the final upstream target. Confidence does not represent "the likelihood that the answer is correct" — it represents the decision service's certainty that "this model is the most appropriate choice." The higher the confidence, the greater the pull of the decision opinion on the final score, but whether the selected model's answer actually meets business requirements still requires independent validation.