GPUStack
An open-source GPU cluster management and inference platform that supports inference backends such as SGLang by containerizing inference backends and providing visual configuration. It offers a web-based interface and enables production-level deployment on the day a model is released.
Timeline (last 90 days)
GPUStack v2.3.0rc2 previews advanced routing capabilities that select the appropriate generative model for requests via the Jev decision service.
Key names in the capability description must match the actual candidate model names exactly; routing aliases or provider names cannot be used, otherwise the mapping to candidates will fail.
The decision contribution weight is the relative influence on the score, not a traffic percentage.
Higress is a cloud-native API gateway open-sourced by Alibaba Cloud, built on Envoy, and GPUStack uses it as the gateway layer for handling AI inference traffic.
Actual request forwarding, plugin execution (including calls to the Jev decision service), and load balancing all occur on the Higress side, while GPUStack is responsible for managing model deployments and routing policy configuration and synchronizing them to Higress.
The only reliable means of verifying routing selection is through log tracing, as the HTTP response itself carries no information about which upstream model generated it.
In testing, all nine inputs of varying difficulty were routed through demo-smart, and the actually selected models matched expectations in every case.
The three tasks of translation, interval merging, and system design were each repeated three times in non-streaming mode with stable model selection, and the streaming versions also selected the same models.
All Facts (8)
The three tasks of translation, interval merging, and system design were each repeated three times in non-streaming mode with stable model selection, and the streaming versions also selected the same models.
50%UnverifiedKey names in the capability description must match the actual candidate model names exactly; routing aliases or provider names cannot be used, otherwise the mapping to candidates will fail.
50%UnverifiedThe decision contribution weight is the relative influence on the score, not a traffic percentage.
50%UnverifiedHigress is a cloud-native API gateway open-sourced by Alibaba Cloud, built on Envoy, and GPUStack uses it as the gateway layer for handling AI inference traffic.
50%UnverifiedActual request forwarding, plugin execution (including calls to the Jev decision service), and load balancing all occur on the Higress side, while GPUStack is responsible for managing model deployments and routing policy configuration and synchronizing them to Higress.
50%UnverifiedThe only reliable means of verifying routing selection is through log tracing, as the HTTP response itself carries no information about which upstream model generated it.
50%UnverifiedIn testing, all nine inputs of varying difficulty were routed through demo-smart, and the actually selected models matched expectations in every case.
50%UnverifiedGPUStack v2.3.0rc2 previews advanced routing capabilities that select the appropriate generative model for requests via the Jev decision service.
50%