The Model Routing Cost Trap: How Retry Costs Devour Your Savings

A Seemingly Clever Cost-Saving Strategy

When building large-scale LLM applications, "model routing" has become almost a standard approach for cutting costs: send simple requests to small models, and reserve large models for complex ones. The logic is intuitive — most everyday requests don't need the capabilities of a top-tier model, and a smaller, cheaper model can handle them just fine, significantly reducing overall inference costs.

Model Routing is an increasingly mature middleware design pattern in LLM application architecture. The core idea is to insert a "router" component between user requests and model inference. This component — typically based on classifiers, embedding similarity, or heuristic rules — evaluates the complexity of a request and decides which model tier should handle it. For example, the price gap between OpenAI's GPT-4o-mini and GPT-4o can be 10-30x. If 70% of everyday requests can be handled by the smaller model, the theoretical cost savings are substantial. The rise of this pattern is closely tied to the explosion of open-source small models (such as Mistral 7B, Llama 3 8B, etc.) throughout 2023-2024 — their performance on specific tasks has become good enough to make tiered routing practically viable.

However, a real-world case shared by a Reddit developer revealed the hidden pitfall behind this strategy: average costs did go down, but p95 tail costs actually went up. This counterintuitive result deserves attention from every team implementing model routing.

The Average Lied: The Hidden Cost of Retries

The core issue is "retry cost." The developer described a typical failure chain:

The requests generated from the router were simple requests sent to the small model. Average cost went down, but p95 cost went up. Most low-confidence requests would first execute on the small model, replay the same context, then fall back to the large model for a second generation.

In other words, when the router judged certain requests as "low confidence," what actually happened wasn't a simple either/or between the small and large model — it was a sequential stacking of "small model + large model." These requests first ran on the small model and failed, then the exact same context was fed to the large model for another run.

It's important to understand the engineering meaning of "confidence" in a router. Confidence typically refers to how certain the routing decision model is about its classification result. Common implementations include: scoring requests with a lightweight classifier (e.g., logistic regression outputting a probability), measuring distance from known patterns in embedding space, or having the small model generate first and then using self-evaluation to judge output quality. When confidence falls below a set threshold, the system triggers a fallback mechanism. The problem is that most fallback implementations use a "try first, then give up" sequential strategy, rather than a "predict and route directly" approach. A more advanced technique is speculative execution — sending requests to both large and small models simultaneously, keeping the small model's result if quality passes and discarding the large model's result — but this introduces additional computational waste and is only worth considering in scenarios where latency is extremely critical.

The result: these requests pay the cost of two generations, and the second one is the most expensive large model call. The average cost gets pulled down by the large volume of successfully handled simple requests, masking the exorbitant tail costs of these "double-billed" requests. This is precisely why p95 (95th percentile) costs spike — it accurately captures the worst routing decisions.

Why Averages Are Misleading

This is a classic lesson from the observability world. When you only watch a single aggregate metric (like average cost), it's easy to develop a false sense of the system's true health. The long tail of the distribution is often where the real problems hide, and averages dilute that tail away. For cost-sensitive LLM production systems, percentile metrics like p95 and p99 are far more diagnostically valuable than averages.

p95 (the 95th percentile) is a commonly used statistical measure for characterizing distribution tails — it means 95% of all observed values fall below that number. In the systems observability world, p95 and p99 latency are more highly regarded monitoring metrics than average latency because they reveal the real experience users face in worst-case scenarios. This philosophy was first popularized by Amazon in their internal service governance — Jeff Bezos famously required teams to focus on p99.9 latency rather than average latency, because the hardest-hit users tend to be high-value customers with the fullest shopping carts and most complex orders. Applying this thinking to LLM cost management, p95 cost precisely exposes the 5% of requests where routing decisions failed the worst — and these requests may account for a disproportionately large share of total costs.

Using Observability to Decompose Cost Attribution

The developer used evaluation platforms like Braintrust to do several critical things:

• Split token attribution and costs by intent: Instead of looking at one overall number, they figured out exactly how much each category of request was costing.
• Clustered expensive routing paths: They identified which intent categories had particularly high fallback rates — these were the ones creating double-billing.
• Compared routing experiments by quality slices: Rather than evaluating with a single blanket quality score, they examined quality performance dimension by dimension for each request type.

Token Attribution refers to the process of precisely attributing the number of tokens consumed by an LLM and the corresponding costs to each individual request, each intent category, or even each user session. Braintrust is a platform focused on LLM application evaluation and observability. It allows developers to log input/output token counts, model selection, latency, and cost for every model call, and supports slicing analysis by custom dimensions (such as intent type, user cohort, routing path). Similar tools include LangSmith (LangChain ecosystem), Helicone, Portkey, and others. The core value of these tools lies in transforming LLM calls from "black-box consumption" into "white-box attribution," enabling teams to manage LLM cost structures with the same granularity that traditional APM (Application Performance Management) brings to microservice monitoring.

The significance of this methodology: it transforms "model routing" from a black-box decision into a measurable, attributable, optimizable engineering problem. Without fine-grained cost and quality attribution, teams simply cannot know where the money is actually being wasted.

Targeted Routing Optimization Strategies

With attribution data in hand, optimization becomes targeted. The developer made two adjustments:

1. Raised the routing threshold for high-fallback intents: For intent categories that frequently fell back from the small model to the large model, they simply raised the decision bar, making it easier for these requests to go directly to the large model and avoiding the futile first attempt.
2. Trimmed redundant context before the second generation: Since fallback is sometimes unavoidable, they stopped passing the full redundant context verbatim to the second call, reducing token consumption on the retry.

The results were a triple win:

• Cost decreased: Targeted optimization brought down costs in the relevant slices;
• Quality held: Overall output quality didn't regress;
• Latency improved: Because fewer requests needed to pay for two generation passes, end-to-end latency actually got faster.

There's No Free Lunch

Of course, the trade-off is clear: more borderline cases now go directly to the large model. This means giving up some costs that the small model could have saved, in exchange for stability in tail latency and tail costs. This is a clear-eyed trade-off, not a pure victory.

The Three-Dimensional Dilemma: Balancing Cost, Latency, and Quality

The question the developer posed at the end is really a shared challenge for the entire LLM engineering community:

When cost, latency, and quality are all moving in different directions, how do you choose your thresholds?

These three dimensions often pull against each other:

• Lower the routing threshold (more requests go to the small model) → average cost drops, but increased fallbacks may push up tail costs and latency;
• Raise the routing threshold (more requests go to the large model) → tail metrics stabilize, but average cost rises;
• Chase maximum quality → almost always means more large model calls, with both cost and latency rising.

Four Actionable Optimization Principles

Drawing from this case, here are four principles to guide threshold selection:

First, set different thresholds for different intents. There is no single globally optimal threshold. High-fallback intents should have more conservative thresholds, while intents where the small model performs reliably can afford to be more aggressive.

Second, factor retry costs into the expected value calculation of routing decisions. When deciding whether to route to the small model, don't just look at the small model's per-call cost. Calculate the expected value: "small model cost × success rate + (small model cost + large model cost) × fallback rate." When the fallback rate exceeds a certain level, routing directly to the large model is actually cheaper.

From a mathematical perspective, this is essentially an expected cost minimization problem. Let Cs be the small model's per-inference cost, Cl be the large model's per-inference cost, and f be the fallback rate for a given intent category. Then: expected cost of routing to the small model = Cs × (1-f) + (Cs + Cl) × f = Cs + Cl × f; expected cost of routing directly to the large model = Cl. The break-even point is Cs + Cl × f = Cl, i.e., f = (Cl - Cs) / Cl. Using GPT-4o-mini ($0.15/million tokens) and GPT-4o ($2.5/million tokens) as an example, the critical fallback rate is approximately 94% — which seems very high. But if you also factor in the implicit cost of double latency on user experience, the practically acceptable fallback rate threshold drops significantly, typically to around 30-40% before it's worth considering direct routing to the large model.

Third, use percentile metrics rather than averages as decision guardrails. Set p95 cost and p95 latency as hard constraints that cannot be breached, then optimize average cost within those bounds. This effectively prevents the long tail from spiraling out of control.

Fourth, establish a continuous evaluation feedback loop. Request distributions drift as the business evolves — today's optimal threshold may be obsolete tomorrow. As demonstrated in this case, treating routing as something that can be repeatedly experimented with and compared across quality slices is the only way to maintain efficiency over the long term.

Distribution Drift is a universal challenge for machine learning systems in production environments, referring to the actual input data distribution shifting away from what was seen during training or tuning over time. In the LLM routing context, this drift may manifest as: new features launching that bring entirely new intent types, seasonal changes in user behavior shifting the complexity distribution of requests, or even model provider API updates changing the small model's performance characteristics on certain tasks. Engineering practices for a continuous evaluation loop typically include: periodically sampling and manually evaluating routing decision quality, setting up automated A/B experimentation frameworks to compare different threshold configurations, and establishing alerting mechanisms for cost and quality (e.g., automatically triggering a review when p95 cost exceeds 120% of the baseline). This aligns with the model monitoring and retraining philosophy in MLOps, except that the monitoring target extends from model prediction quality to routing decision quality.

Conclusion

The biggest takeaway from this case isn't that "model routing is bad" — it's that any optimization must be built on a foundation of observability. Model routing is an effective cost-reduction technique for LLMs, but without fine-grained cost and quality attribution, it's easy to be blinded by averages while silently bleeding money in the invisible tail.

Break the system apart, attribute by dimension, guard with percentiles — this methodology applies to cost optimization in virtually any LLM production system. In the three-way game of cost, latency, and quality, there are no silver bullets — only continuous, data-driven trade-offs.