vLLM Semantic Router Releases Decision 2.0: Answer 64 Questions Per Request in a Single Inference Pass

What's New in vLLM Semantic Router's Decision 2.0

The vLLM ecosystem has a notable new addition — Semantic Router's Decision 2.0 is officially released. This version delivers significant upgrades to semantic understanding for request routing, with a standout capability: answering up to 64 discriminative questions about a single request in one forward pass, providing dense semantic signals for downstream routing decisions.

For teams building large-scale LLM services, request routing is far from trivial. Requests vary enormously in complexity, intent, sensitivity, and resource consumption. The ability to make fine-grained semantic judgments about a request before it ever reaches the main model enables smarter compute allocation, better model path selection, and a more balanced trade-off between cost and quality. Decision 2.0 is built precisely for this use case.

An Open Model Family Spanning 0.6B to 27B

One of the most noteworthy aspects of this release is the complete open model series ranging from 0.6B to 27B parameters. This range means users can flexibly choose between lightweight and large models based on their latency requirements, hardware constraints, and accuracy expectations.

A model as small as 0.6B is well-suited for fast request classification in edge or high-concurrency scenarios, adding virtually no extra inference overhead. Meanwhile, the 27B-scale model can deliver higher discriminative accuracy when precision matters most. This tiered design means Semantic Router is no longer a one-size-fits-all solution — it can adapt to everything from early-stage startups to large-scale production environments.

Why Answering 64 Questions in One Forward Pass Matters

In traditional approaches, evaluating a request across multiple dimensions — such as whether it involves code, requires internet access, contains sensitive content, or has high complexity — typically requires either multiple model calls or maintaining multiple separate classifiers. Decision 2.0 compresses all of these judgments into a single forward pass, producing answers to 64 questions at once.

The direct benefits of this design are clear: dramatically lower inference costs, more predictable latency, and consistent multi-dimensional judgments. For online services demanding low-latency routing decisions, eliminating even one extra model round-trip means a shorter overall response chain and higher throughput.

From a technical standpoint, "answering multiple questions in a single forward pass" is essentially a variant of multi-label classification. The model simultaneously outputs 64 binary or multi-class classification heads at the final layer, sharing the underlying Transformer representations rather than running a separate full inference pass for each question. This shared-representation approach works well because different judgment dimensions — such as "does this involve code?" and "how complex is this?" — often rely on similar semantic features. Reusing intermediate-layer computations saves FLOPs and implicitly enforces semantic consistency across dimensions, avoiding the logical contradictions that can arise between independently trained classifiers. By contrast, the traditional "serial multi-classifier" approach not only stacks latency linearly but also suffers from inconsistent training data distributions across classifiers.

Apache-2.0 License and Native Transformers Support

On the engineering side, Decision 2.0 is released under the Apache-2.0 license, which is highly permissive for commercial use. Organizations can confidently integrate it into their own products without worrying about licensing restrictions.

Models can also be loaded directly via Hugging Face's Transformers library, lowering the barrier to adoption — developers can plug it into existing inference pipelines without learning any proprietary framework. Combined with vLLM's strengths in high-throughput inference, the coordination between Semantic Router and the main inference engine becomes even smoother.

Implications for LLM Service Architecture

From a broader perspective, Decision 2.0 reflects a clear trend: LLM services are evolving from a "single large model handles everything" approach toward layered, routing-based architectures. Using a lightweight semantic routing layer to evaluate and triage requests before they reach the expensive main model reduces costs while improving overall service quality.

The value of semantic routing solutions like this becomes increasingly apparent as LLM applications scale. The project is led by Xunzhuo Liu and team, and the openly available model weights under a permissive license provide a solid foundation for the community to further research and improve routing strategies.

For teams building production-grade LLM systems, this direction is worth watching closely — a well-designed semantic routing layer may offer better cost-efficiency than simply scaling up to a larger model.

This "pre-processing routing layer + main inference engine" architecture is commonly referred to in engineering as the Router-as-a-Service or LLM Gateway pattern. The core idea is to have a lightweight, efficient routing model perform semantic analysis before requests reach the main model, then dispatch requests to different backends accordingly — for example, routing simple questions to smaller models to reduce cost, triggering safety filters for sensitive content, and prioritizing requests requiring real-time information to internet-enabled paths. This pattern mirrors the API Gateway concept from microservices architecture, with the key difference that routing logic shifts from URL rules to semantic vectors and classification outputs. As enterprise-scale LLM deployments grow, the quality of the routing layer directly impacts overall system cost-efficiency, making it one of the important research directions in the current LLMOps landscape.