Hierarchical Routing
A scheduling strategy that dynamically assigns requests of varying complexity to LLMs of different scales. Using lightweight classifiers or cascading mechanisms, it substantially reduces inference costs while maintaining overall output quality.
Timeline (last 90 days)
Common implementations of tiered routing include rule-based length/perplexity thresholds, using small classifiers to predict query complexity scores, and cascade routing (where a weak model attempts first and then decides whether to escalate).
Tiered routing refers to using cheaper models to handle simple, high-frequency initial screening, and escalating complex or high-risk changes to more capable models.
A model with slightly lower accuracy but five times cheaper may outperform on the valid findings per dollar metric, provided tiered routing is used to cover high-risk scenarios.
All Facts (3)
Common implementations of tiered routing include rule-based length/perplexity thresholds, using small classifiers to predict query complexity scores, and cascade routing (where a weak model attempts first and then decides whether to escalate).
50%UnverifiedTiered routing refers to using cheaper models to handle simple, high-frequency initial screening, and escalating complex or high-risk changes to more capable models.
50%UnverifiedA model with slightly lower accuracy but five times cheaper may outperform on the valid findings per dollar metric, provided tiered routing is used to cover high-risk scenarios.
50%