Reasoning Depth, Latency, and Cost: A Design Guide Based on Escalation-Based Routing

Why a Single-Model Strategy Is Doomed to Fail

Most teams building LLM applications fall into the same trap: they first get the workflow running with the strongest model, then discover that latency and bills are both spiraling out of control. A request to look up an exchange rate and a compliance analysis requiring multi-step policy reasoning consume the same computational resources.

This design guide proposes a clear path out: replace one-size-fits-all model selection with Escalation-Based Routing, so that each request receives only the reasoning depth it truly needs.

Diagnose First, Prescribe Second: Five Typical Symptoms

Before rebuilding the architecture, you need to locate which layer the problem actually lies in. The guide provides a diagnostic table mapping common symptoms to likely root causes and inspection items:

• Very low latency but vague answers: Usually the model's own reasoning depth is insufficient; check whether the chain-of-thought steps or policy details the request requires are being satisfied.
• Latency spikes but cost remains controllable: A small model is being forced to retry repeatedly or degrade into heuristic rules; review token usage and the number of inference calls.
• High cost per thousand tokens but acceptable latency: Most likely a large model is handling simple tasks; verify the "simple" mapping in your routing rules.
• Inconsistent policy compliance across similar requests: Retrieval context is missing or stale; confirm the retrieval component is pulling the latest authoritative sources.
• Frequent escalation to the largest model: The reasoning budget thresholds are defined too loosely, or risk thresholds are set too low.

The core value of this table is that it breaks down the vague feeling of "the model isn't performing well" into specific, locatable, fixable steps. Symptoms are clues, not proof of root cause—before blaming the model's pre-training capabilities, first check label quality, missing context, and the preprocessing pipeline.

Decision Logic of the Three-Tier Escalation Architecture

The heart of the entire architecture is a simple decision flow:

1. Is the request a "simple task"?—Single query, no multi-step reasoning required, deterministic rule validation
2. If yes, route to the small model; if not, check whether the latency budget is sufficient
3. If the latency budget is sufficient, route to the mid-tier model; if not, escalate directly to the large model

The key concept here is the Reasoning Budget: defined jointly by three dimensions—token count × cost × maximum latency—and exposing a "budget exceeded" flag.

The model pool needs at least three tiers: small model (fast, cheap, limited depth), mid-tier model (balanced latency and reasoning capability), and large model (highest reasoning depth, higher cost). Each tier is exposed through a unified inference API and reports its own token rate and unit price.

LLM Routing is the broader concept behind this architecture: dynamically selecting the most appropriate model among multiple candidates based on request characteristics, rather than being fixed to a single model. Common routing strategies in the industry include: rule-based static routing (hardcoded by keyword or request type), classifier-based learned routing (training a small model to predict request difficulty), and the escalation-based routing adopted in this article (starting from the lowest-cost tier and ascending level by level when the budget is exceeded). The advantage of escalation-based routing is its extremely low decision cost (just one lightweight judgment) and its inherent "fail-safe" property—in the worst case, the request will ultimately land on the large model and won't be permanently denied a high-quality answer due to a classification error. The cost is that complex requests may incur the latency overhead of two inference passes, so latency budget estimates need to leave headroom.

Responsibility Boundaries of Six Core Components

Architectural clarity depends on clarity of responsibility boundaries. This design splits the system into six key components:

Request Router is responsible for classifying incoming payloads (simple vs. complex) and selecting the initial model tier based on the decision flow. It consumes request metadata and consults the reasoning budget engine.

Reasoning Budget Engine calculates the budget for the current tier—token × cost × maximum latency—and exposes a "budget exceeded" flag. It reads pricing tables, latency SLAs, and request complexity estimates.

Cost/Latency Monitor records the actual token consumption, wall-clock latency, and monetary cost of each request, and feeds data back to the router for adaptive throttling. It triggers alerts when SLA drifts.

Model Pool holds at least three model variants and exposes a unified inference API.

Retrieval Layer provides external up-to-date documents (RAG) only for context that needs a factual grounding, but must never be allowed to replace deterministic rule evaluation.

Deterministic Rules Engine runs outside the LLM, enforcing business-critical policies that must be 100% correct (such as compliance thresholds). The LLM can suggest rules, but the engine is responsible for validating them.

Additionally, an optional Fine-Tuning / LoRA Service steps in only when prompt engineering and retrieval prove insufficient, training low-rank adapters. Adapters are frozen after evaluation passes, and only merged for inference when regression tests pass.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Rather than modifying the base model's original weights, it injects a pair of small low-rank matrices (A and B) into the Transformer's attention layers, updating only these two matrices during training. Since the parameter count is typically only 0.1%–1% of the original model, LoRA dramatically reduces VRAM usage and training costs, and allows training multiple adapters for different tasks that can be loaded or switched on demand at inference time. It must be emphasized that LoRA freezes the base weights, but the behavioral changes introduced by the adapter may still affect the model's performance outside the target task, which is why the regression testing step in the "train—evaluate—freeze—regression test—merge" pipeline cannot be omitted.

RAG (Retrieval-Augmented Generation) refers to retrieving relevant document snippets from an external knowledge base before generating an answer, and concatenating them as context into the prompt. It allows the model to access factual information beyond its training data with greater timeliness, and is therefore widely adopted in frequently changing scenarios like exchange rates and policy clauses. But one must distinguish "factual grounding" from "factual correctness": RAG merely hands candidate evidence to the model—whether the model faithfully cites it, whether the retrieval results themselves are authoritative, and whether the documents are stale are all outside RAG's guarantees. This is also the fundamental reason the article stresses that real-time values must go through a separate API rather than being cached into the prompt—the retrieval layer cannot replace deterministic value retrieval for requirements with hard timeliness constraints.

Two Key Architecture Decision Records

ADR 001: Adopt Escalation-Based Model Selection

Context: The platform handles both high-throughput queries and low-throughput strategic analysis. Early prototypes used a single large model, resulting in unacceptable latency for most traffic and ballooning compute costs.

Decision: Adopt a three-tier escalation strategy (small → mid → large), governed by the reasoning budget engine. The router initially classifies each request as "simple" or "complex" based on lightweight static analysis (token estimates, presence of chain-of-thought clues). If the budget is exceeded after the first inference, the system automatically re-routes to the next tier.

Acceptance Criteria: 95% of requests complete within the latency SLA of their assigned tier; average daily compute cost is reduced by ≥ 30% compared to the single-model baseline; policy violation incidents do not increase after introducing the escalation mechanism.

ADR 002: Keep Real-Time Value Retrieval Outside LLM Inference

Context: Some fraud detection requests require the latest exchange rates. An early design cached exchange rates into the prompt to avoid extra calls, causing occasional compliance errors when the cache expired.

Decision: Introduce a tool/API component that the LLM invokes via function-calling-style prompts. This component fetches values from authoritative services at request time, and the results of real-time value calls are never cached.

Acceptance Criteria: Each real-time value request triggers exactly one external API call; auditable logs show the fetch timestamp matches the request timestamp within ±5 seconds; no compliance test failures due to stale data after launch.

Teaching Example: Design Rationale of the Python Decision Assistant

The guide provides a teaching-oriented Python module demonstrating budget calculation and routing logic. The core function _budget_exceeded checks three conditions simultaneously: whether the cost obtained by multiplying estimated token count by unit price is within threshold, whether the actual latency meets the tier SLA, and whether the token count exceeds the model's context window. Only when all three conditions are satisfied does it refrain from escalating.

The decide_model function starts from "simple requests begin with the small model," measures elapsed time, and escalates to the large model if the budget is exceeded. All configuration constants (cost, latency limits, token limits) are placeholders; production systems should inject real values from a centralized policy store.

The engineering value of this code lies not in its logical complexity, but in turning the fuzzy judgment of "when to escalate" into three quantifiable, configurable, testable conditions.

Reasoning Budget differs from a traditional cost budget in that it is a multi-dimensional constraint rather than a single monetary cap. The article defines it as the joint constraint of token count, monetary cost, and maximum latency, meaning a request is only considered "within budget" when all three dimensions are satisfied simultaneously. The engineering significance of this design is that latency and cost often trade off inversely (a large model is more expensive but may get it right in one pass; a small model is cheaper but may retry repeatedly and end up slower), and looking at any single metric leads to bad decisions. Packaging the three into a boolean flag "budget exceeded" simplifies complex multi-objective optimization into a testable escalation trigger condition—which is also why the sample code _budget_exceeded checks all three conditions at once.

Multi-Source Consensus on Evaluation and Maintenance

Regardless of which party's practice guide you reference, the following inspection items recur:

• Unit tests for decide_model must cover simple/complex, budget-exceeded, and time-boundary scenarios
• End-to-end routing integration tests using simulated latency and cost curves
• The regression test suite for deterministic rules engine output must remain unchanged after any LoRA update
• Verify that the per-tier averages reported by the cost/latency monitor match the SLA table
• Sample retrieval-augmented prompts to verify that the factual grounding is consistent with source documents
• Quarterly check of whether the reasoning budget engine's thresholds still align with business risk tolerance

The guide specifically emphasizes several editorial accuracy principles that form a consensus across different sources: RAG improves factual grounding but does not guarantee correctness, completeness, or compliance; LoRA freezes the base model weights and learns low-rank updates, but does not guarantee that all prior behaviors are preserved, requiring regression testing outside the target task; real-time value APIs must be called at request time, and cached or stale values are unacceptable in any compliance-sensitive operation.

What This Design Actually Solves

The essence of escalation-based routing is turning "model capability" from a global configuration item into a resource dynamically allocated per request. By explicitly defining escalation paths, quantifying reasoning budgets, and separating deterministic logic from probabilistic inference, teams can simultaneously maintain low latency, costs proportional to a request's true difficulty, and alignment between capability and task complexity.

This is not a solution chasing peak performance, but an engineering framework pursuing controllability. Its value truly emerges at scale—when processing millions of requests daily, a 30% cost reduction and a 95% SLA attainment rate matter far more than a few extra percentage points of accuracy on individual difficult tasks.