动态模型路由

LLM traffic in production is unevenly distributed: a large number of requests are simple deterministic transformations (such as field extraction, classification, formatting), and only a small portion requires frontier models to handle.

When migrating requests with a high cache hit rate to local models, the actual comparison benchmark should be the frontier model's cache read rate rather than the standard input price; ignoring this layer leads to flat spending but worse latency.

Classification routing should not be based on caller identity (team or user), because the same team submits both trivial requests and difficult requests; routing by caller optimizes the wrong variable and leads to quality complaints.

The escalation rate is a first-order term in the cost model: if the cheap tier costs C1 per call, the frontier tier costs C3, and the escalation rate is r, the effective cost is approximately C1 + r·C3; when r rises from 5% to 40%, the latter term may dominate or even exceed the baseline of sending everything to the frontier tier.

Routing without an escalation path is not routing but downgrading; each tier needs explicit failure conditions (such as schema validation failure, low confidence, refusal, timeout) to promote requests to a higher tier.

A router that misclassifies 40% of traffic downward may be more expensive and slower than having no router at all, because escalated requests pass serially through two tiers, the end-to-end latency is the sum of the two, and the p95 latency is composed of these requests.