The Four Real Culprits Behind Your Soaring LLM Bill: Don't Rush to Swap in a Cheaper Model

The Bill Went Up, But Traffic Barely Moved

The invoice arrived again, more expensive than last month. Traffic grew only a little, yet the bill jumped sharply. Most teams' first instinct is "we need to switch to a cheaper model" — sometimes that's right, but more often the money is leaking from places nobody is watching, and switching models just moves the leak somewhere else.

A developer community author broke the problem down into four suspects: long prompts, verbose output, retries and agent loops, and model-tier mismatch. He argues that the first two can be spotted just by looking at a single request, while the latter two — retry loops and model mismatch — only surface when you look at the entire workflow.

Suspect A: Long Prompts — Paying for the Same Text on Every Call

Input tokens are usually cheaper per unit than output tokens. The problem is that you pay for them on every single request. A 6,000-token system prompt, if it never changes, costs 6,000 tokens on the first call and still costs 6,000 tokens on the millionth call.

The key to saving money is putting the stable part of the prompt at the top, so the vendor's prompt caching mechanism kicks in. Both OpenAI and Anthropic document this: when the prompt prefix is exactly the same, cached input tokens are read at a discount.

A few things to check:

• Is the stable system prompt placed first so the cache can hit?
• Does your RAG retrieve 20 chunks when the answer only needs 3?
• Is the conversation history trimmed or summarized, or does it grow without bound?

Suspect B: Verbose Output — Burning Money on the Most Expensive Side of the Price Sheet

On most vendors' price sheets, output tokens cost several times more than input tokens. A model that "helpfully" writes 600 words when the UI only displays two sentences is burning your most expensive token type.

For most applications, this is the easiest problem to find and fix:

• Have you set a reasonable max_tokens for each use case?
• Does the prompt explicitly state the length and format you want?
• For machine-consumed output, are you asking directly for structured JSON instead of letting it write a long essay that you then parse?

Output length is fundamentally a product decision, not a model capability issue. Many teams default to "let it run free" and end up paying the price in the part of the bill that's hardest to explain.

Suspect C: Retries and Agent Loops — Paying Repeatedly for the Same Work

This is the one that really hides. A single user action can fan out into: a re-prompt triggered by a validation failure, a retry triggered by a vendor timeout, an agent that calls a tool, gets a confusing result, and tries again — eight calls in total.

Each step looks perfectly reasonable on its own. Added together, total cost is several times the budget.

How to investigate:

• Calculate cost by "completed task" rather than "request." If a "summarize ticket" task takes four LLM calls on average, then four is your true unit price.
• Set hard caps on agent iterations and retries per request.
• Confirm whether retries are idempotent and logged, so you can see exactly how many times they fire.

Tracing tools make these hidden costs visible. Langfuse's token and cost tracking can attribute cost by trace and span; if you route through the LiteLLM proxy, it can also give you spend statistics by key, team, or model. Without observability, retry loops are an invisible line item.

LLM tracing is the concrete form of observability in AI applications. Traditional APM treats an HTTP request as the smallest unit, whereas a single user action in an LLM app fans out into multiple layers of calls: a single agent turn may include planning, tool calls, reflecting on results, and re-planning — several steps, each of which may trigger internal retries. Tracing organizes these calls into a tree-structured trace and span hierarchy, where each span records the model name, input and output token counts, latency, and status, making "how much did this task cost" queryable. Langfuse, LangSmith, Phoenix, and similar tools all provide this kind of capability.

The LiteLLM proxy, meanwhile, sits in the routing layer between your application and the various model vendors. It unifies different APIs like OpenAI, Anthropic, and Bedrock into an OpenAI-compatible format, while centrally logging usage and cost for every call, and supporting budget and rate limits by API key, team, and model. When your application forwards through it, hidden retries, fallback calls after timeouts, and automatic downgrades across multiple models all show up in the same ledger — which is exactly what turns an "invisible bill" into "attributable spend."

Suspect D: Model-Tier Mismatch — Using a Sledgehammer on Every Nail

A huge number of calls in production are actually boring: classification, field extraction, ticket routing. These tasks rarely need the strongest model you have access to.

How to investigate:

• List all LLM call sites and tag each one: reasoning-intensive or routine.
• For routine tasks, run a smaller model against an evaluation set. If quality holds, switch.
• For latency-insensitive work, look at batch endpoints — OpenAI Batch and Anthropic Message Batches are both priced below their respective synchronous APIs.

OpenAI's latency optimization guide makes a related point: fewer tokens, fewer requests, and using smaller models when possible improves speed while also controlling spend. Speed and cost are not in conflict here.

The essential difference between batch endpoints and synchronous APIs is trading latency for price. Synchronous calls require the model to return results immediately, consuming real-time compute; batch processing queues requests and lets the vendor execute them opportunistically within a several-hour window (usually 24 hours), so unit prices are commonly cut in half or even lower. OpenAI Batch and Anthropic Message Batches both follow this pattern. The tradeoff is that low latency isn't guaranteed, making them suitable for offline evaluation, data labeling, batch summarization, log cleanup, and other time-insensitive work. Two things to watch when using them: first, requests must be packaged and uploaded in formats like JSONL with custom IDs for matching results back; second, failed items are returned separately, so you need to design rerun logic on the business side to avoid partial failures being silently swallowed.

A 30-Minute Audit You Can Run Right Now

1. Pull a day's worth of traces or logs.
2. Group cost by task, not by request.
3. For the three costliest tasks, record: input tokens, output tokens, calls per task, and the model used.
4. The biggest number on that table is your answer to A, B, C, or D.

Nine times out of ten, the solution isn't switching vendors — it's a cap, a cache-friendly prompt ordering, or a smaller model on the routine path. What these four suspects share is that none of them is a model capability problem; they're all engineering habit problems.

Frequently Asked Questions

How much can prompt caching save? It depends on the prefix reuse rate. When the prefix is exactly the same, both OpenAI and Anthropic offer discounted pricing for cached input tokens. The longer the stable prefix and the more frequent the calls, the bigger the savings.

How do I know when to switch to a smaller model? Run your evaluation set against candidate smaller models. Routine tasks (classification, extraction, routing) are usually affected the least — if quality holds, you can switch.

How do I control the cost of agent loops? Set hard caps on iteration counts, measure cost by completed tasks, and plug in tracing tools like Langfuse or LiteLLM to make retries visible.

One-line Summary

Soaring bills are usually not about an expensive model, but about four kinds of engineering habits that leak cost.

Paragraph Summary

The article breaks down an anomalous LLM bill into four suspects: long prompts, verbose output, retries and agent loops, and model-tier mismatch. The first two can be spotted by looking at a single request — stable prompt prefixes should go at the top to hit the cache, and output should use a reasonable max_tokens with structured JSON instead of prose. The latter two are subtler and only surface when you look at the whole workflow: a single task can fan out into eight calls due to validation failures, timeouts, or tool confusion, so cost should be measured by "completed tasks" rather than "requests," with hard caps on iterations and retries; meanwhile, many routine tasks (classification, extraction, routing) could be handled by smaller models or batch endpoints. The author offers a 30-minute audit: group a day's cost by task and find the input/output token counts and call frequencies of the three costliest ones — the answer emerges on its own. The solution is usually caps, cache-friendly prompt ordering, or smaller models along the path — not switching vendors.