AI Agent Token Cost Optimization: Three Levers to Cut LLM Token Spend

Most of your AI Agent bill is input tokens—and it's spent on "re-reading." Every turn, a coding agent resends the full conversation transcript to the model; combined with repository exploration and tool outputs, a single session can carry a hundred thousand tokens of context, read over and over at full price. The original author runs a whole fleet of overnight coding agents on a single machine and feels this cost pressure firsthand. They've distilled three levers already running in production: summarize with a cheap model first, put expensive reasoning at both ends of the V-model, and route requests through a proxy based on price.

One caveat up front: the author is candid that these mechanisms were measured, but end-to-end savings were not accounted for—the arithmetic in this article is based on public prices and their own logs, so treat it as illustrative rather than a real bill.

Lever One: Retrieve First, Let a Cheap Model Summarize

The most expensive way to give a frontier model context is to dump raw search results into it; the cheapest way is to let a small model read them first.

The pattern works like this: a local search tool retrieves candidate snippets from the codebase or knowledge base; before those snippets enter a GPT-6- or Fable-class session, a low-cost model synthesizes them into a compact answer with deterministic citations that trace back to the source. The expensive model receives the summary plus a handful of citations it may need to reopen—not ten near-duplicate excerpts.

The author uses the open-source tool terraphim-grep, which resolves query terms through a curated knowledge-graph thesaurus and then ranks local matches. The --answer flag requests a generated answer when a sufficiency check calls for one, and --force-rlm forces the synthesis path; only the selected snippets leave the machine, sent via OpenRouter to a DeepSeek^Ad-flash-class model. There's also a --search-only mode that disables synthesis entirely and skips building the LLM client, so even if an API key lingers in the environment, no network call is made.

In their validation run, the raw snippet content was 505 characters, the evidence with source locations was 1108 characters, and the synthesized answer text was 440 characters—the frontier model received roughly 40% of the original context, with citations it can reopen on demand.

Two honest caveats here. First, this measures characters, not tokens; the CLI doesn't expose token usage, so the ratio is only a directional reference. Second, the benefit depends on the handoff shape: the full answer object with all ten citations measured 1918 characters, and if forwarded as-is, the savings evaporate. Pass only the synthesized result plus the citations needed for verification—nothing else.

Lever Two: Put Expensive Reasoning at Both Ends of the V-Model

The second lever isn't a tool but a lifecycle decision: match model price to work stage.

The author organizes agent work into a V-model. The left side asks "what should we build" (research, design, spec); the right side asks "did we build it right" (review, verification, judging panels). The long middle stretch (implementation, writing tests, mechanical verification loops) is where the most tokens burn, and it's precisely the part that needs the least deep reasoning.

So models are split accordingly:

• Expensive models at both ends. Research synthesis, architecture decisions, and final review stay on high-reasoning channels, because a bad plan or a rubber-stamped review costs more than the tokens saved.
• Fast delivery models in the middle. Implementation and test loops run on grok, MiniMax, or Kimi highspeed channels via flat-rate subscriptions. Overnight agents execute on opencode-go subscription models; judging panels run on Kimi K2.5, completing a review in about 9 seconds with 90% agreement with human reviewers.

The arithmetic is worth doing once. At public API prices, an illustrative workload (100M input tokens, 20M output tokens) run entirely on a frontier model costs $800; routing 80% of it to a mid-tier model costs $480 for the same token volume—40% saved, with the same number of models. This is a counterfactual estimate at public prices, not a measured bill, but the shape of the savings is real: you're not paying for fewer tokens, you're paying a lower unit price for stages that don't need frontier reasoning.

The catch is discipline. Left to its own devices, an agent will use the same model from start to finish. Stages must be explicitly defined, so this lever only works within a structured lifecycle—not in free-running sessions.

V-Model originates from systems engineering and software engineering; it's a lifecycle model that pairs each development stage one-to-one with a verification stage. The left descending branch runs through requirements analysis, system design, architecture design, and detailed design, while the right ascending branch runs through unit testing, integration testing, system testing, and acceptance testing, forming a V shape overall. Its core idea: every development stage should have a corresponding verification stage, and verification criteria should be defined during the development stage. In this article's context, the author borrows the V-model to emphasize that both the left side ("what to build") and the right side ("did we build it right") require deep reasoning, while the implementation and test loops at the bottom of the V consume the most tokens yet demand the least reasoning depth—making them suitable for cheap, fast models.

Lever Three: Put a Proxy in Front of the Providers

The third lever makes the cheap route the default: place an LLM proxy between the agent and each provider, routing by request content or by each channel's current speed.

The author's proxy runs on localhost, exposes OpenAI- and Anthropic-compatible endpoints, and decides where each request goes:

• Keyword routing. The proxy uses Aho-Corasick string matching (microsecond-scale overhead, the same crate as the knowledge graph) to match prompt content against a markdown taxonomy. A prompt saying "think step by step about this architecture" gets routed to the think chain, regardless of what model ID the client requested. In their logs, three consecutive requests across three scenarios landed on three different providers, completely transparently to the client.
• Speed and price routing. Candidate channels can be ordered by time-to-first-token or by cost tier: free first, then subscription, then pay-as-you-go. Each scenario can choose latency priority or price priority for itself. The idle time an agent spends waiting on rate limits is a real cost—just denominated in hours rather than tokens.
• Fallback chains make cheap channels safe. Cheap providers do have reliability gaps: silent API format changes, undocumented capacity errors, higher latency when accessed from Europe. Pipe-delimited fallback chains mean that if one provider goes down, the request moves to the next target. The author verified this through deliberate fault injection (pointing the primary provider's domain to localhost in /etc/hosts) rather than trusting the config.

One caveat that surprised them: prompt caching is per-model, not per-session. Measured in real coding sessions, when the model switched mid-session, 82% of the time cache reads dropped below 10% of the previous value: a warm 107000-token prefix became 110000 tokens of uncached input, with the same text priced at roughly ten times the cached rate. So the savings pattern is routing per session or per scenario, not randomly switching every turn. Pick a channel for the task and stay on it.

Aho-Corasick algorithm is a multi-pattern string matching algorithm proposed by Alfred Aho and Margaret Corasick in 1975. By building a finite state automaton (similar to a trie with failure pointers), it can match multiple keywords simultaneously in a single scan, with a time complexity of O(n + m + k), where n is the text length, m is the total pattern length, and k is the number of matches—independent of the number of patterns. This makes it well suited for low-overhead keyword recognition in request routing: the proxy can determine in microseconds whether a prompt contains trigger phrases like "think step by step," and thus decide the routing target. Compared with matching regexes one by one, Aho-Corasick has a clear advantage when there are many patterns, and it's simple to implement with no external dependencies.

Prompt Caching is an optimization mechanism offered by several LLM providers: when a request's prefix (such as the system prompt or a long conversation history) exactly matches a previous request, the provider can reuse the already-computed key-value cache (KV Cache), avoiding recomputation and thereby reducing latency and input token costs. Typically, cache-hit tokens are priced at only about one-tenth of uncached tokens, but the cache has a time-to-live (from minutes to hours) and is isolated per model. This means the same conversation history cannot share a cache across different models. In this article, the 82% collapse in cache reads occurs precisely because switching models mid-session forces the new model to recompute the entire prefix from scratch, turning previously cheap cached tokens into full-price input tokens.

What the Author Doesn't Claim

Token savings were not measured end-to-end. The synthesis example measures characters, not tokens; the model-mix example is arithmetic at public prices, not an invoice; the 82% cache-collapse figure comes from one operator's session logs, not a cluster benchmark. The honest metric is "quality-adjusted cost per accepted task," and they're still building the measurement infrastructure to report it.

What they can claim is the mechanisms: retrieval that filters before an expensive model reads, a lifecycle that spends reasoning budget where judgment lives, and a proxy that prices channels before forwarding requests. None of these require trusting a benchmark, and all three are running today.

Full articles on the related components are available: terraphim-grep and role-aware search, and The V-Model for AI Agents. The grep tool lives in the terraphim-clients repository.