How to Cut LLM API Costs by 70% Without Touching Code

A developer recently went public with their LLM API bill optimization experience: with user volume and request volume essentially unchanged, the monthly spend dropped from $212 to $61—about a 70% reduction. The key was that they didn't rewrite the application logic, only adjusted the call structure at the API layer. The original author broke down the entire method in their blog.

LLM API cost optimization in practice

Defaulting to flagship models is the root cause of runaway costs

When most developers integrate an LLM, their first instinct is to "pick the strongest one." The author's initial architecture was exactly that: from simple email classification to full document rewriting, every request hit the most expensive endpoint.

The price gap is an order of magnitude:

GPT-4o: input $5.00 / 1M tokens, output $15.00 / 1M tokens
Claude 3.5 Sonnet: input $3.00 / 1M, output $15.00 / 1M
Smaller models (e.g., GPT-4o-mini): input $0.15 / 1M, output $0.60 / 1M

Input costs differ by 30x, output by 25x. After reviewing, the author found that about 70% of calls were simple classification, entity extraction, or short-text summarization—tasks that smaller models can handle, with lower latency to boot.

LLM API billing uses tokens as the base unit, which can be roughly understood as the text fragments the model processes. In English, 1 token is about 0.75 words; in Chinese, about 1.5 to 2 characters. Pricing splits into input (prompt) and output (completion), with output typically 3 to 5 times more expensive because autoregressive generation requires repeated forward computation per token. This means input and output must be costed separately: a 1000-token input and a 1000-token output do not produce the same bill. In addition, context window size and whether a model supports structured output also affect real-world usability and hidden overhead. Understanding token economics is the measurement basis for all the optimization strategies that follow—tiering by task, setting max_tokens, and trimming system prompts are all essentially about reducing unnecessary token consumption.

Strategy 1: Route models by task tier

The author split all API calls in the application into three tiers:

Simple tasks: keyword extraction, sentiment analysis, tag generation, short translations
Medium tasks: summarization, code explanation, structured data extraction
Complex tasks: long-form writing, multi-step reasoning, generating code from scratch

Only the third tier hits the flagship model endpoint. In implementation, it's just a simple configuration mapping:

class ModelRouter:
    ROUTES = {
        "classify": ("openai", "gpt-4o-mini", 50),
        "extract": ("openai", "gpt-4o-mini", 200),
        "summarize": ("anthropic", "claude-3-5-haiku", 500),
        "code_review": ("anthropic", "claude-3-5-sonnet", 1500),
        "long_form": ("openai", "gpt-4o", 4000),
    }

    @staticmethod
    def get_endpoint(task_type: str):
        return ModelRouter.ROUTES.get(
            task_type,
            ModelRouter.ROUTES["long_form"]
        )

The function signature stays the same, and the call site is still call_llm("summarize", text), but the underlying endpoint and parameters have switched. This one item alone cut about 35% of the bill—the largest effect of all the strategies.

Strategy 2: Semantic caching, not just exact matching

Using Redis caching at the application layer is common, but the author points out an overlooked fact: many requests are semantically identical but textually different. When users ask about refund policies, they might say "can I get a refund," "how do refunds work," or "money back process"—each of these previously hit the LLM once.

He added a lightweight semantic cache in front of the LLM: each prompt first goes through an embedding, and if its cosine similarity to a historically cached prompt exceeds a threshold (he used 0.92) and the model matches, the cached result is returned directly.

def llm_with_cache(prompt, task_type, threshold=0.92):
    emb = client.embeddings.create(
        input=prompt,
        model="text-embedding-3-small"
    ).data[0].embedding

    for cached_prompt, cached_response in cache.items():
        if np.dot(emb, cached_prompt) > threshold:
            return cached_response
    response = call_llm(task_type, prompt)
    cache[tuple(emb)] = response
    return response

Embedding calls cost far less than LLM calls, and a single cache hit saves $0.01 to $0.50. The author's data showed that about 30% of duplicate user queries hit the cache, further reducing overall spend by about 20%.

Embedding is a technique that maps text into high-dimensional vectors, where semantically similar sentences are closer together in vector space. The system uses an embedding model (such as text-embedding-3-small) to convert prompts into vectors, then uses cosine similarity to compare the cosine of the angle between two vectors, typically ranging from -1 to 1, with values closer to 1 indicating greater semantic similarity. A threshold of 0.92 means that two requests, even if worded differently, are treated as the same type of query as long as their semantics are highly close, allowing direct reuse of the cached result. There's a tradeoff here: too high a threshold misses reusable requests, while too low a threshold may misjudge semantically different questions as the same and return the wrong answer. Therefore the threshold needs to be tuned according to business tolerance, with a small amount of manual spot-checking to validate cache hit quality. Embedding calls are typically one to two orders of magnitude cheaper than LLM calls—this is the economic premise that makes semantic caching viable.

Strategy 3: Don't pay for tokens you don't use

This optimization was the most embarrassing for the author. He previously set no output constraints for classification tasks, and the model would churn out three paragraphs when all that was needed was a "yes" or "no" label.

The fix was straightforward: explicitly set max_tokens for each task type in the routing configuration, and use JSON mode or structured output constraints wherever possible. On a flagship model, a 200-token output and a 2000-token output differ in cost by about 10x, while the logical result is completely identical. This item alone saved $20 to $30 per month.

Strategy 4: Price arbitrage across multiple providers

Different providers price equivalent-quality models noticeably differently. The author's comparison:

Provider / Model	Input (per 1M)	Output (per 1M)	Notes
OpenAI GPT-4o	$5.00	$15.00	Strong overall capability
Anthropic Claude 3.5 Sonnet	$3.00	$15.00	Better long-context handling
Google Gemini 1.5 Pro	$3.50	$10.50	Strong coding ability
Small and mid-size providers	$0.15–$1.00	$0.60–$5.00	Sufficient for most tasks

The author found that for about 80% of the workload, using mid-tier models and alternative providers made no difference to user satisfaction at all. He set up a fallback chain: prefer the cheaper provider first, fall back to the flagship model on error or timeout. The application code stays the same—just swap the API base URL and add a wrapper layer.

He also mentions that there are now unified API gateways that can dynamically switch providers. In the original article, he recommended an on-demand, pay-as-you-go gateway service he uses (tai.shadie-oneapi.com), declaring it for personal use rather than sponsored.

An API gateway is a middle layer between the application and various large-model APIs, exposing a unified request format externally while handling authentication, routing, rate limiting, retries, and format conversion internally. The application only needs to point its base URL at the gateway, without writing different SDK code for each provider. When a provider times out or errors, the gateway can automatically forward the request to an alternative model, enabling the fallback chain mentioned in the article. Such gateways typically charge by call volume or take a cut, and some offer unified billing and usage dashboards. When using a third-party gateway, be aware of data passing through a third party, vendor lock-in, and compliance risks—especially in scenarios involving sensitive data; building your own lightweight wrapper or using a cloud provider's built-in managed gateway are also common alternatives.

An often overlooked item: invisible overhead in the context window

The author additionally warns of an easy pitfall: overly long system prompts. If the system prompt is 500 tokens while the actual input is only 50 tokens, that means 91% of the money is spent on boilerplate. He audited all system prompts, removed duplicate content, moved static instructions to the application layer, and reduced base prompts by 40%.

Where the 70% comes from

The author's cost breakdown:

Task-model matching: about 35%
Semantic caching: about 20%
Token budget trimming: about 10%
Multi-provider price differences: about 5%

This method introduces no new frameworks and refactors no business code. The core actions are just three: routing, caching, budget control. For applications with monthly LLM spend above $50, the author's judgment is: spend an afternoon on the audit and tiered routing, and a 70% cost reduction is reproducible in most production applications that rely heavily on LLMs.

The LLM API market's pricing is still fragmented, and the habit of defaulting to the strongest model won't disappear anytime soon—but the bill will make the decision for you.

One-line Summary

A developer cut monthly LLM spend from $212 to $61 without changing business code, using only routing, caching, and budget control.

Paragraph Summary

Without rewriting application logic, a developer cut their monthly LLM bill from $212 to $61—about a 70% reduction—just by adjusting the API call structure. The core insight: about 70% of calls are simple classification, extraction, and summarization tasks that don't need a flagship model. Specific strategies include tiered model routing by task complexity (~35% savings), a semantic cache in front of the LLM to catch duplicate queries (~20%), setting max_tokens and structured output per task to avoid paying for useless tokens (~10%), and using multi-provider price differences to build a fallback chain (~5%). He also notes that overly long system prompts are hidden overhead, trimming base prompts by 40% after an audit. The whole methodology comes down to three actions: routing, caching, budget control.