Inference Networks: Optimizing LLM Inference Costs with Threshold Policies

Large language models (LLMs) have become the default choice for natural language processing tasks, but their steep inference costs make the cost-performance trade-off an unavoidable practical concern. A recent paper published on arXiv introduces a graph-based framework called Inference Networks, offering a theoretically optimal solution for how to combine and deploy multiple expert models.

The Pain Point of Multi-Model Collaboration: No Principled Approach

In real-world deployments, engineers often orchestrate multiple expert LLMs — either in an ensemble (parallel voting) or in series (cascaded calls). Intuitively, routing simple queries to cheaper, smaller models while reserving powerful but expensive models for complex ones is clearly more economical. This is the essence of adaptive routing.

The problem is that the industry has long lacked a systematic method to answer the question: "How should we optimally use the models we have?" When should you escalate to a more expensive model? What criteria should trigger that escalation? Most practice relies on intuition and trial-and-error rather than provably optimal strategies. This paper targets exactly that gap.

Inference Networks: Abstracting Model Combinations as Graphs

The paper's core contribution is abstracting the multi-model coordination problem into a graph-based framework. In an Inference Network, nodes represent different LLMs and edges represent conditional model activations — meaning a downstream model is only invoked when certain conditions are met.

The "Inference Network design problem" is then to determine the optimal topology — i.e., to find the model usage pattern that best balances cost and performance. This graph perspective is valuable because it unifies previously scattered strategies — routing, cascading, ensembling — under a single mathematical framework, making "optimal topology" a formally defined and solvable optimization problem.

Graph structures are not new to machine learning, but applying them to describe model invocation topology is the paper's key innovation. Traditional cascade systems use a fixed linear chain; a graph structure allows richer topological relationships — for example, a node branching to multiple downstream models under different conditions, or outputs from multiple models converging before a final decision. The key word in "conditional model activations" is conditional: each edge carries a triggering rule, and only when that rule is satisfied (e.g., confidence below a threshold, predicted class belonging to a subset) does the downstream model get called — avoiding unnecessary computation. This design naturally supports dynamic computation paths. Within the same batch of queries, different inputs may activate entirely different subsets of models, allowing overall cost to adapt to input difficulty.

Key Finding: Optimal Policies Have Threshold Structure

The paper starts with the most fundamental topology — a series of chained LLM experts, each with different costs and different levels of expertise (characterized by model confidence). The problem is formalized as: minimize expected inference cost subject to a target performance constraint.

For this class of Inference Networks, the paper delivers an elegant theoretical result — the optimal activation policy has a threshold structure:

• Query the lowest-cost LLM first;
• Only call a more expensive model when that model's confidence falls below a set threshold.

The significance of this result is that it reduces a seemingly complex combinatorial optimization problem to a simple, executable rule: just compute a few thresholds.

Discriminative Tasks vs. Generative Tasks

The paper further distinguishes the threshold form for two types of tasks:

• Discriminative tasks: The optimal policy consists of a set of thresholds — one per class;
• Generative tasks: The optimal policy requires only a single threshold.

This distinction is practically useful. Classification tasks have discrete output spaces with noticeably different confidence distributions across classes, requiring finer per-class threshold control. Generative tasks can typically be measured by a unified metric, making a single threshold sufficient.

From Theory to Practice: Threshold Computation and Confidence Estimation

Theoretical proofs alone aren't enough — the paper also provides two key engineering components:

1. A structured method for computing thresholds: How to derive concrete threshold values from cost and performance constraints;
2. Confidence estimation mechanisms: Actionable confidence estimation schemes for both discriminative and generative tasks.

These two components determine whether the framework can actually be used in production. After all, the effectiveness of any threshold policy depends entirely on the reliability of confidence estimates — if a model assigns high confidence to its incorrect answers, the threshold mechanism breaks down.

Confidence calibration is the central challenge here. Whether a model's confidence score genuinely reflects the probability of being correct directly determines whether the threshold strategy works. Research has shown that large neural networks commonly suffer from overconfidence — even when predictions are wrong, softmax output probabilities can approach 1. In practice, post-processing techniques like temperature scaling and Platt scaling are commonly used to recalibrate confidence scores, making them statistically closer to true accuracy. For generative tasks, defining confidence is inherently more complex; common approaches include averaging log-probabilities over generated sequences, majority voting across multiple samples (self-consistency), or training dedicated verifier models to assess output quality. The paper's task-specific solutions are a direct response to this engineering complexity.

Experimental Results: Significant Cost Reduction While Staying Within Performance Budget

The paper validates its approach through experiments on open-source LLMs. Results show that the optimal threshold strategy achieves substantial cost reductions while satisfying pre-specified performance budgets.

This means that rather than always calling the most powerful and expensive model, a cheaper model can "go first," and only when it is "unsure" does the system escalate. For enterprises that call LLMs at scale and high frequency, this cascade strategy delivers real inference cost savings without sacrificing overall quality.

Value and Limitations

The greatest value of this work is elevating "multi-model cascading" from engineering folklore to a theoretically grounded practice — the proof of threshold structure's optimality lets practitioners execute the rules confidently, without endless hyperparameter tuning.

That said, the paper's current theoretical results primarily cover the basic "series expert" topology. Optimal policies for more complex Inference Network topologies — such as hybrid structures with branching and parallel ensembles — remain open questions. Furthermore, the robustness of confidence estimation under real-world distribution shift is a critical area to validate before production deployment. Overall, Inference Networks provide an elegant and practical new perspective on LLM cost optimization.

"Distribution shift" refers to the phenomenon where data encountered during deployment differs in distribution from training or calibration data. For Inference Networks, this means the historical performance data used to compute thresholds may no longer represent future queries, causing preset thresholds to become suboptimal — queries that should escalate to a stronger model may get intercepted by the cheap model due to inflated confidence scores. In practice, this requires periodically recalibrating thresholds or introducing online monitoring to detect performance degradation. Additionally, the current framework assumes model costs are fixed and known, but real-world cloud API pricing (per token, per request, with concurrency discounts, etc.) is far more complex. Incorporating dynamic pricing into the optimization objective is another practical challenge for bringing this framework to production.