Practical AI Cost Optimization: A Deep Dive into Intelligent Routing and Context Compaction

Why Your AI Bills Keep Growing

As LLM applications move from prototypes to production environments, many teams start facing a very real problem: AI usage costs are ballooning fast. Every conversation, every Agent task, every round of tool calls accumulates Token consumption, ultimately showing up on the monthly bill.

Tokens are the basic units that large language models use to process text. They don't map directly to characters or words—for English, one Token is roughly equal to four characters; for Chinese, one character typically corresponds to one to two Tokens. This difference stems from the underlying mechanics of tokenizers: mainstream LLMs generally use Byte Pair Encoding (BPE) or its variants (such as SentencePiece and tiktoken) for tokenization.

BPE was originally proposed by Philip Gage in 1994 for data compression. After being introduced to the field of neural machine translation in 2016, it became the standard tokenization approach for today's mainstream large language models. Starting from single characters, the BPE algorithm repeatedly merges the most frequently occurring adjacent byte pairs in the corpus, eventually forming a vocabulary that covers high-frequency words and common subwords. Its core advantage lies in gracefully handling the out-of-vocabulary (OOV) problem: even when encountering a brand-new word never seen before, it can be broken down into a combination of known subwords. OpenAI's tiktoken tokenizer, developed specifically for the GPT series, is written in Rust and runs 3-6 times faster than the Python implementation; its cl100k_base vocabulary contains about 100,000 Token units. Chinese, due to the high semantic completeness of individual characters and its different combinatorial nature compared to English letters, is inherently less encoding-efficient in BPE vocabularies than English—this is the fundamental reason why Chinese prompts with the same semantic content often consume more Tokens than English ones. The practical significance of understanding this mechanism is that word choice, format arrangement (such as Markdown heading symbols), and number notation in prompts all affect the Token count, which in turn directly affects cost.

Mainstream model providers (such as OpenAI and Anthropic) all use a bidirectional pricing model of "input Tokens + output Tokens." Take GPT-4o for example: input pricing is about $2.5/million Tokens, while GPT-4o mini is only about $0.15/million Tokens—a gap of more than 16 times. This means model choice itself is the most critical variable in the cost structure.

Even more tricky is that many teams, in pursuit of quality, habitually send all requests to the most powerful and most expensive model. But in reality, not every task requires a top-tier model—a simple classification or formatting task can be completed with high quality by a small model, at perhaps one-tenth the cost of a large model, or even less.

Current mainstream LLM providers have all developed clear tiered product matrices, providing a natural space for cost optimization. Take OpenAI for example: the product line spans dozens of times in price difference, from the lightweight GPT-4o mini to the flagship o1-pro. Anthropic's Claude series similarly has three tiers: Haiku (fast and cheap), Sonnet (balanced), and Opus (top-tier). Google's Gemini Flash series pushes the cost-performance ratio of lightweight models to new heights. According to multiple public cases, a reasonable routing strategy can reduce enterprise AI usage costs by 40% to 70%, while keeping the impact on overall task completion quality within an acceptable range.

This article, based on a practical tutorial shared in the Reddit community, outlines two AI cost optimization strategies that can be implemented without large-scale refactoring: Intelligent Routing and Context Compaction. What they have in common is that they deliver results quickly, require minimal changes, and offer clear returns.

Intelligent Routing: Finding the Most Suitable Model for Each Request

Core Idea: On-Demand Allocation, Goodbye to One-Size-Fits-All

The essence of intelligent routing is to dynamically assign requests to the most suitable model based on their complexity and type, rather than sending everything to the most expensive option indiscriminately. The core value of this strategy lies in: using cheap models to handle simple tasks and reserving expensive compute for the complex scenarios that truly need it.

The key to implementation is building an LLM Gateway. Engineering-wise, an LLM Gateway is essentially a reverse proxy service, whose architectural pattern is derived from API gateway designs in microservice ecosystems, borrowing from mature patterns of traditional API gateways like Kong and Nginx, but deeply extended for the special needs of large models. Unlike traditional API gateways, an LLM Gateway must additionally handle engineering challenges unique to large models: transparent forwarding of streaming responses (Server-Sent Events), queuing and merging of asynchronous batch requests, error code mapping between different providers (e.g., OpenAI's 429 rate-limit error differs in format from Anthropic's overload response), and precise tracking of Token billing. Core functional modules typically include: a unified interface adaptation layer (standardizing different providers' API formats into an OpenAI-compatible protocol), authentication and key management (centrally managing API keys from various providers, supporting rotation and quotas), request logging and cost tracking (recording Token consumption item by item, supporting allocation by project or user), and rate limiting and circuit breaking (preventing a single downstream anomaly from causing cost runaway or service avalanche).

There are already several open-source implementations in the industry. Among them, LiteLLM is an open-source LLM Gateway with over 15,000 stars on GitHub, supporting 100+ model providers. Its core design philosophy is to abstract all models into a unified OpenAI-compatible interface, so switching from GPT-4 to Claude or Gemini only requires changing the model parameter, with zero changes to business code. PortKey builds on this by strengthening observability and workflow orchestration capabilities. In enterprise scenarios, some teams also choose to build a lightweight routing layer on infrastructure like AWS API Gateway or Cloudflare Workers to gain more fine-grained access control and auditing capabilities.

This gateway acts as a unified entry point for all model calls—your Agent no longer calls a specific model's API directly, but first sends the request to the gateway, which decides which model to actually route to. The advantage of this decoupled design is obvious: routing strategies can be flexibly adjusted without modifying business code.

Using a Prompt Classifier to Make Routing Decisions

The core component of routing decisions is the Prompt Classifier. After a request enters the gateway, it quickly determines the type and complexity of the prompt, then selects the corresponding target model.

The prompt classifier itself is typically driven by a lightweight language model or a rule engine. In practice, there are multiple implementation paths, presenting clear trade-offs across three dimensions: latency, cost, and accuracy. Rule-based approaches (keyword matching, regular expressions, request-length thresholds) are the simplest to implement, with P99 latency usually under 1 millisecond, but they struggle with semantic ambiguity—for example, "help me write a piece of code" and "help me write a poem" have identical structures, so the rule layer cannot distinguish the difference in complexity. Embedding-similarity approaches encode the request into a vector, then compute cosine similarity against representative vectors of predefined categories, with latency around 10-50 milliseconds and medium accuracy, suitable for business scenarios with clear category boundaries.

Specially trained small routing models represent the most refined approach currently available. The open-source RouteLLM framework, developed by a Stanford University research team in collaboration with Anyscale, systematically demonstrated in its 2024 paper that a router trained on RLHF preference data can maintain overall task scores above 95% of GPT-4's level while routing 50% of requests to GPT-3.5-tier models. The commercial product Martian adopts an online learning mechanism, where the router continuously updates its decision boundaries based on actual task completion feedback, achieving automatic optimization over time. It's worth noting that the router's own inference latency needs to be controlled within 5% of the routed task's P50 latency; otherwise, for low-latency scenarios (such as real-time conversation), the extra time consumed by the classifier would offset the value of the cost savings.

In practice, the Routing Table is the most intuitive management tool. It uses prompt type and complexity as dimensions and explicitly maps them to corresponding models. A typical tiered routing logic looks like this:

• Simple tasks (classification, format conversion, intent recognition) → small, cheap models
• Medium tasks (general Q&A, summarization, structured extraction) → mid-tier models
• Complex tasks (multi-step reasoning, code generation, complex planning) → flagship-tier models

This routing table transforms LLM cost-reduction strategies from a vague "gut feeling" into executable, maintainable engineering rules. When a new model launches or prices change, you only need to update the routing table—no need to touch the Agent's core logic.

Context Compaction: Putting Conversation History on a Diet

The Hidden Cost Trap of Long Sessions

During Agent operation, especially in scenarios with long conversations or multi-round tool calls, context keeps accumulating. Each round of calls re-stuffs the entire history into the request, while Token billing is calculated based on the total of inputs and outputs. This means—the longer the conversation, the more each call is paying extra for an increasingly bloated history.

Behind this linear or even superlinear growth lies clear mathematical logic: assuming each round of conversation adds 500 Tokens, by round 10 the input size of a single request is already 10 times that of round 1. If a flagship model is used to handle a hundred-round long session, the cumulative cost of history Tokens could far exceed the inference cost of the actual task itself. It's worth noting that even though some current models have expanded their context windows to 1 million Tokens, a larger window does not mean a free lunch.

Stanford University's 2023 paper Lost in the Middle revealed a critical flaw in the attention mechanism: when key information is placed in the middle of an ultra-long context, the model's accuracy drops by more than 20 percentage points compared to when the information is at the beginning or end. This phenomenon stems from inherent biases in Transformer positional encoding and attention distribution—the model has a natural tendency to focus on the beginning and end of the context, while information in the middle section is easily "drowned out." This means that even when using Gemini 1.5 Pro, which supports a 1-million-Token context, blindly piling up historical records not only wastes cost but may also degrade reasoning quality because key information is ignored. Actively managing context and explicitly placing important information at the beginning or end of the prompt is a dual means of improving quality and reducing cost. Context Compaction is precisely the solution to this pain point.

Three Mainstream Compaction Implementation Patterns

The idea of context compaction originates from text summarization techniques in the field of natural language processing, but it has taken on new engineering significance in the era of large models. Early solutions were simple sliding-window truncation—keeping only the most recent N rounds of conversation—but this causes the Agent to lose early key information. Modern compaction strategies are more refined; their design inspiration partly comes from the "limited working memory capacity" theory in cognitive science—recent information is preserved as high-precision original text, earlier information is converted into semantic summaries, and the oldest information retains only key facts or is fully archived to an external vector database (such as Pinecone or Weaviate) and recalled on demand during retrieval. Common implementation patterns include:

• Rolling Summary: When conversation history exceeds a set length, earlier parts are summarized into concise summaries, replacing the lengthy original content. This pattern draws on the human memory model of "working memory + long-term memory," preserving recent information with high fidelity and compressing older information as semantic summaries. Some frameworks (such as LangChain's ConversationSummaryBufferMemory and Mem0) have built this mechanism in as a standard component, and mainstream Agent frameworks like LangGraph and AutoGen also provide tiered memory support to varying degrees.
• Key Information Extraction: Retains only the context fragments truly useful for the current task, discarding irrelevant intermediate processes.
• Tiered Memory: Recent messages retain their original text while older messages are stored in compressed form, forming a "hot-warm-cold" three-tier memory system.

Worth mentioning is that Anthropic has introduced a native Prompt Caching feature in the Claude series of models, whose working principle comes directly from the underlying characteristics of the LLM inference architecture. During Transformer inference, each input Token needs to compute attention weights with all other Tokens and generate the corresponding Key-Value matrix (KV Cache), which is the main component of GPU inference cost, with time complexity of O(n²). When multiple requests share exactly the same prefix, the server side can persist the KV Cache corresponding to that prefix to high-speed storage, and subsequent requests directly load the existing cache without recomputation. Anthropic's Prompt Caching requires the cached prefix to contain at least 1024 Tokens, with a cache storage duration of 5 minutes, and once hit, the input Token cost drops to 10% of the original price. OpenAI's similar mechanism (Cached Input Tokens) has also been rolled out in the GPT-4o series, automatically taking effect for repeated prefixes exceeding 1024 Tokens, with no explicit configuration required by developers. The two mechanisms complement each other—compaction reduces the total number of Tokens, while caching reduces the unit price of repeated Tokens, simultaneously lowering costs from both the total-volume and unit-price dimensions.

The implementation of compaction strategies needs to be clear and controllable, letting developers know exactly which information is retained and which is compressed. This is crucial—overly aggressive compaction can cause the Agent to "lose its memory," which in turn affects task quality. Therefore, while optimizing Token consumption, one needs to find a balance between cost savings and information completeness.

The Combined Value of the Two Strategies

Intelligent routing and context compaction are worth deploying together because they tackle the cost problem from two different dimensions:

• Routing solves "which model to use"—letting cheap models handle appropriate work and lowering the cost of a single call.
• Compaction solves "how much content to send"—reducing the number of Tokens and shrinking the scale of each call.

Combining the two can achieve considerable overall cost reduction without significantly sacrificing quality. More importantly, both strategies require no large-scale refactoring of the Agent: routing is transparently inserted at the architectural level via the LLM Gateway, while compaction is progressively introduced as a conversation-management module.

Implementation Recommendations: Advance in Phases

For teams looking to control AI spending, we recommend proceeding in the following order:

1. Build the LLM Gateway first: Converge all model calls into a unified entry point to lay a solid foundation for subsequent strategies. It's advisable to first evaluate open-source solutions like LiteLLM and PortKey to avoid reinventing the wheel.
2. Then add classification and routing tables: Start diverting the most obvious simple tasks and gradually refine routing rules. In the early stages, you can begin with a rule-based classifier, then introduce embedding similarity or specially trained semantic classification models (referencing open-source frameworks like RouteLLM) after accumulating data, finding a balance between accuracy and latency that suits your business.
3. Finally introduce context compaction: For long-session scenarios, test the impact of different compaction strategies on cost and quality. At the same time, pay attention to models' native caching features (such as Claude's Prompt Caching and OpenAI's Cached Input Tokens), which complement compaction strategies to simultaneously lower Token costs from both the total-volume and unit-price dimensions.
4. Continuously monitor and iterate: Observe the actual cost distribution of various types of requests, and use the data to reverse-optimize routing tables and compaction thresholds.

It should be noted that the content of this article comes from a single community tutorial. Specific routing table configurations and compaction parameters need to be tested and tuned according to your own business scenarios and should not be copied directly. The core logic of AI cost optimization remains consistent—spending money where it matters most, while ensuring quality.