vLLM Semantic Router
A semantic routing component based on the vLLM inference engine that places a lightweight routing layer in front of the inference engine. It dispatches incoming requests to the most appropriate downstream model or service based on the semantic content of each request, with the goal of reducing average inference cost.
Timeline (last 90 days)
The vLLM Semantic Router team released a family of decision models called Decision 3.0 (abbreviated as d3), covering multiple parameter scales from 0.6B to 27B, released under the Apache-2.0 open-source license, and supporting three input modalities: text, image, and video.
vLLM Semantic Router released Decision 3.0, a set of multimodal decision models licensed under Apache-2.0
vLLM Semantic Router places a lightweight routing layer in front of the vLLM inference engine, using a decision model to determine the request type and direct it to a specialized model
A semantic routing architecture can route high-frequency simple requests to small models and complex requests to large models, creating a dynamic balance between cost and quality
This project is led and developed by Xunzhuo Liu and his team.
Using an efficient semantic routing layer to make decisions and route traffic before requests reach the large model can both reduce costs and improve service quality.
vLLM Semantic Router released Decision 2.0.
All Facts (7)
A semantic routing architecture can route high-frequency simple requests to small models and complex requests to large models, creating a dynamic balance between cost and quality
50%UnverifiedvLLM Semantic Router places a lightweight routing layer in front of the vLLM inference engine, using a decision model to determine the request type and direct it to a specialized model
50%UnverifiedThis project is led and developed by Xunzhuo Liu and his team.
50%UnverifiedUsing an efficient semantic routing layer to make decisions and route traffic before requests reach the large model can both reduce costs and improve service quality.
50%UnverifiedThe vLLM Semantic Router team released a family of decision models called Decision 3.0 (abbreviated as d3), covering multiple parameter scales from 0.6B to 27B, released under the Apache-2.0 open-source license, and supporting three input modalities: text, image, and video.
50%UnverifiedvLLM Semantic Router released Decision 3.0, a set of multimodal decision models licensed under Apache-2.0
50%UnverifiedvLLM Semantic Router released Decision 2.0.
50%