LLM Routing
An optimization strategy for LLM serving that uses a complexity estimator to automatically assess the difficulty of user queries, routing simple queries to lower-cost, smaller models and complex queries to more capable, larger models in order to balance service cost and output quality.
Core Facts
Timeline (last 90 days)
Escalation-Based Routing is an LLM routing strategy that starts from a low-cost tier and escalates level by level when the reasoning budget is exceeded.
Optimal LLM routing depends on accurately modeling prefill cost and queue latency; putting cache hit benefits and queuing cost into the same cost function gets closer to low latency than optimizing either one alone.
Routing an LLM by total prompt length (directing requests below approximately 500 characters to the economy tier and above 500 to the premium tier) causes short prompts carrying tool definitions to be incorrectly routed to lower-capability models.
Prompt length measures cost, not capability requirements; these are two fundamentally different dimensions.
Traditional LLM routing typically decides which model to use based solely on the individual request itself (e.g., text length, complexity, or domain classification), making it a stateless, isolated judgment.
All Facts (7)
LLM routing is an inference optimization strategy that dynamically selects models based on task complexity, cost budget, and response speed requirements; open-source frameworks such as RouteLLM and LiteLLM are already exploring this direction.
65%UnverifiedLLM路由领域的学术研究可追溯到2023年的多篇论文
80%UnverifiedEscalation-Based Routing is an LLM routing strategy that starts from a low-cost tier and escalates level by level when the reasoning budget is exceeded.
50%UnverifiedOptimal LLM routing depends on accurately modeling prefill cost and queue latency; putting cache hit benefits and queuing cost into the same cost function gets closer to low latency than optimizing either one alone.
50%UnverifiedRouting an LLM by total prompt length (directing requests below approximately 500 characters to the economy tier and above 500 to the premium tier) causes short prompts carrying tool definitions to be incorrectly routed to lower-capability models.
50%UnverifiedPrompt length measures cost, not capability requirements; these are two fundamentally different dimensions.
50%UnverifiedTraditional LLM routing typically decides which model to use based solely on the individual request itself (e.g., text length, complexity, or domain classification), making it a stateless, isolated judgment.
50%