Routing Tax
The cost incurred for providing a unified interface and dynamic routing, including classifier latency and compute overhead for every request, as well as potential single points of failure and observability blind spots.
Timeline (last 90 days)
Layered routing saves the integration tax (a unified API key, base URL and billing relationship) but adds a routing tax (roughly 2 milliseconds of classifier latency plus compute overhead).
The integration tax grows linearly with the number of models, while the routing tax is usually fixed and predictable, so layered routing is more economical when managing a large number of models.
All Facts (2)
Layered routing saves the integration tax (a unified API key, base URL and billing relationship) but adds a routing tax (roughly 2 milliseconds of classifier latency plus compute overhead).
50%UnverifiedThe integration tax grows linearly with the number of models, while the routing tax is usually fixed and predictable, so layered routing is more economical when managing a large number of models.
50%