Cost Per Correct Answer Is a Trap: Why It Always Recommends the Cheap Model

In LLM production systems, the trade-off between accuracy and cost is almost inevitable, but the common "cost per correct answer" metric can mislead you into choosing the cheapest model rather than the solution that truly meets your quality requirements.

The Flaw in the Metric Itself

A simulation of "cost per correct answer" over 2,000 requests shows: a small model (1 unit cost/request) achieves 74.5% accuracy with a cost per correct answer of 1.34; while a large model (20 units) achieves 95.8% accuracy with a cost per correct answer of 20.88. By this metric, the small model wins decisively.

This is no coincidence. In any scenario, as long as the small model barely suffices, this metric will award it the victory. The author's initial draft even asserted the opposite conclusion, but an in-page check overturned that assumption.

Set the Quality Floor First

The question that actually has an answer is: given a promised accuracy, what is the minimum cost to achieve it?

For example, when promising 90% accuracy, the small model is immediately disqualified—not because it is cheap, but because it does not qualify. At this point, a cascade strategy with a verifier costs only 28% of what always using the large model would cost. When promising 95%, difficulty-based routing costs 32%. The entire value of routing is that savings only become visible once you write down the quality threshold.

The difference between Cascade and Routing: A cascade is a sequential execution strategy where requests first go to a low-cost model, and if its confidence or a verifier deems the result unreliable, the same request is escalated to a stronger model for a retry, so each request may involve multiple calls. Routing is a front-loaded decision that selects a model before the call based on estimated difficulty (or metadata), typically making only one call. The sources of savings differ: cascades amortize cost by ensuring "most requests don't escalate," while routing saves money by "simple requests never calling the large model at all." In this article, the cascade cost under a 90% quality promise is 28%, and the routing cost under a 95% promise is 32%—these are precisely the two mechanisms' performance at different quality thresholds.

The role of the Verifier: The verifier here is not a human reviewer, but an automatically runnable correctness judgment function or lightweight model used within the cascade to determine whether the small model's output meets the bar. Its presence introduces additional cost and latency, but can significantly reduce the number of erroneous requests flowing to the large model. If the verifier itself is not reliable enough, the cascade's cost-saving effect quickly disappears.

The Trap of Confidence-Based Routing

Letting the small model escalate when uncertain seems reasonable. But the data shows: when the small model's confidence is 90%, its actual accuracy is 0%. Confidence measures fluency, not correctness. There exists a range where the model is 90% confident yet 0% correct, and the escalation mechanism that should cover these requests fails due to the confidence design.

LLM Confidence and Calibration: A language model's output confidence typically comes from token probabilities, self-reported probabilities, or consistency across repeated samples, but these signals measure "how fluent or concentrated the generation is," not "factual correctness." A model can generate very confident statements about completely false facts, because wrong answers are equally frequent and equally coherent in the training distribution. Calibration refers to how well a model's self-reported confidence aligns with its actual accuracy; an uncalibrated model will exhibit a range where it is "90% confident yet 0% correct." Therefore, using confidence directly as a routing or escalation trigger amounts to making cost decisions on a wrong proxy metric, ultimately neither saving money nor meeting the quality floor.

Routing Itself Has a Cost

Most comparisons ignore the router's overhead. It is also a model call, billed on every request. In code implementations, if the router cost is not included, the overhead table is meaningless.

Why router cost is often ignored: In comparison tables in papers or blog posts, the router is often abstracted as a free function, but in production implementations, judging input difficulty itself usually requires a model call (e.g., a classifier, embedding similarity, or lightweight LLM scoring). This overhead is billed per request and is directly added to the total cost. If the router cost is c_r, the small model cost is c_s, and the large model cost is c_l, then the true expected cost of the routing strategy is c_r + p·c_s + (1-p)·c_l, where p is the proportion going to the small model. When c_r approaches or even exceeds c_s, the seemingly significant savings are greatly eroded. Therefore, any routing scheme must, before deployment, incorporate the router's own call count, token consumption, and latency into the same bill.

Ceiling and Reproducibility

An oracle router knows in advance which requests the small model will get wrong, sending only erroneous requests to the large model; its accuracy matches always using the large model, at only 29% of the cost. Difficulty-based routing achieves 32%—a small gap.

All results use no API and no sampling, with the difficulty function being u² (u uniformly distributed), so every number can be reproduced via closed-form solution. 184 verifier assertions, 34 in-page checks, 0 failures.

Oracle Router: This is a theoretical upper bound that knows in advance whether each request will be answered incorrectly by the small model, thereby sending only doomed requests to the large model and handling the rest with the small model. It does not represent any deployable system, but is used to measure how much room for improvement remains in real routing strategies. In this article, the oracle router costs 29% and difficulty-based routing costs 32%—close enough to indicate that the difficulty function has already captured most of the predictable error signals. If the gap between real routing and the oracle is large, it means the current routing features (such as input length, domain tags) are insufficient to distinguish hard from easy requests, and further router optimization yields limited returns—shifting to cascades or verifiers would be better.

Conclusion

Used in isolation, "cost per correct answer" rewards the cheapest model that barely suffices. The correct framework is to fix the quality floor first, then find the most economical path that meets it. The savings from strategies like routing and cascading only become apparent once the quality threshold is explicit.

One-line Summary

"Cost per correct answer" misleads you into choosing the cheapest model; the right approach is to set a quality floor first, then find the lowest cost.

Paragraph Summary

The article points out that the common metric "cost per correct answer" has a structural flaw: in a 2,000-request simulation, the small model achieves 74.5% accuracy with a cost per correct answer of 1.34, while the large model achieves 95.8% accuracy with a cost per correct answer of 20.88—the metric unhesitatingly picks the small model. But the question that actually has an answer is: given a promised accuracy, what is the minimum cost to achieve it? Once a 90% quality floor is set, the small model is immediately disqualified, and a cascade with a verifier costs only 28% of always using the large model; when promising 95%, difficulty-based routing costs 32%. The article also warns that the small model's confidence is decoupled from correctness, with a range where it is 90% confident yet 0% correct, and that the router's own call cost is often ignored. All results are reproducible via closed-form solution with no API dependency.