Local Code Indexing: Cutting AI Coding Token Costs by 94%

It Starts With a Bill

Developer Raj and his friend Foss were working on a project together, using a whole suite of AI coding tools like Claude Code, Cursor, Copilot, and Codex every day. One month the bill looked normal; the next month it suddenly spiked. The project hadn't changed, the tools hadn't changed—the only difference was that they'd been using them more.

When they dug into it, they discovered a counterintuitive fact: most of the cost wasn't going toward the AI's "thinking," but toward sending too much useless context—files the model didn't need at all, irrelevant code, sent repeatedly with every single query.

They ran a real test: on their own project, a typical query sent 45,000 tokens of context, but only about 5,000 tokens were actually useful. The remaining 40,000 tokens were worthless, yet they had to pay for them on every query. In Raj's words, it was like "ordering one pizza every time but paying for the other nine you'll never eat."

Token Billing Mechanics and the Economics of the Context Window: A token is the fundamental unit that large language models use to process text, and understanding its billing logic is the first step in controlling costs. Roughly speaking, each English word corresponds to about 1–2 tokens, while Chinese characters typically map to 1–2 tokens each. Mainstream AI coding tools (like Claude, GPT-4, etc.) charge separately for input tokens and output tokens, and the price gap between the two is enormous—take Claude 3.5 Sonnet as an example: input costs around $3 per million tokens, while output is as high as $15. Although output is more expensive per unit, in practice input volume tends to far exceed output, making the input side the real cost driver. More importantly, every API call counts the entire context window content toward the input cost—even if your question is just 10 words long, if you attach ten thousand lines of code as context, you pay the full price for those ten thousand lines.

There's an architectural trap here that's easy to overlook: the larger the context window, the worse the waste. GPT-4 Turbo supports 128K tokens, and Claude 3.5 supports up to 200K tokens. This expansion brings greater processing power, but it also has a counterproductive effect—the larger the context window, the easier it is for AI tools to automatically fill it up. Modern AI coding tools (like Claude Code and Cursor) automatically grab currently open files, project configuration, recent conversation history, and more to fill the context whenever a query is issued. This process is invisible to the user, yet it often results in large amounts of redundant information being silently sent. The bigger the window, the more auto-filled content, and the higher the bill climbs. In the information retrieval field, this phenomenon is called "context pollution"—useful information is diluted by large amounts of irrelevant content, which not only drives up costs but also degrades the model's reasoning quality.

Three Failed Attempts

Before finding the real solution, they tried three approaches. The first two failed.

Attempt One: Optimizing the Prompt

They wrote "Please be concise, only show relevant code" in the prompt. It sounds reasonable, but it simply doesn't work—because before the model even reads that instruction, those 45,000 tokens of context have already been sent. The cost is incurred before the prompt is even read.

Attempt Two: Adjusting Model Parameters

They tried tweaking settings like max tokens and temperature. Same problem: these parameters only affect output, but the money is spent on input.

It's worth clarifying what these two parameters actually do: max_tokens limits the maximum length of a single model reply, while temperature controls the randomness and creativity of the output (0 for deterministic output, 1 for highly random). Both act on the model's generation phase and have no effect whatsoever on input tokens that have already been sent. This is a cognitive misconception many developers fall into—thinking that adjusting these parameters will "save money," when in reality it only affects output quality, not the cost structure.

Attempt Three: Compressing the Output

This approach did have some effect. They had the model write shorter answers, reducing output volume by 75%. But here's the key: output only accounts for about 10% of total cost. Cutting a small number by 75% still leaves you with a small number—nowhere near enough.

The Key Conclusion: The Money Is on the Input Side

This is the most central insight of the entire analysis:

• Input (files, search results, context) accounts for about 90% of total cost
• Output (the code the AI writes back) accounts for only about 10%

This means:

• Cutting output by 75% saves only about 8% of total cost
• Cutting input by 94% saves about 61% of total cost

Same math, wildly different results. The conclusion is clear: if you want to fix it, fix the input side—that's where the token dollars are flowing away.

The Solution: A Local Code Search Layer

They built a search layer that runs locally, sitting between the codebase and the AI. Instead of dumping entire files into the AI, they let the AI query an index and retrieve only the small snippet of code it actually needs.

The Evolution of RAG Architecture in Code Scenarios: This local code search layer is essentially a concrete implementation of the Retrieval-Augmented Generation (RAG) architecture in a code scenario. RAG was proposed by Meta AI Research in 2020, with the core idea of dynamically injecting retrieval results from an external knowledge base into the large model's context, allowing the model to access the latest or proprietary knowledge without retraining.

RAG has gone through several generations of evolution since its introduction: from the initial single-vector retrieval (Naive RAG), to Advanced RAG which introduced reranking models, to Modular RAG capable of multi-step reasoning. In the codebase scenario, RAG faces unique challenges: code's semantic units (functions, classes) have clear boundaries, making them suitable for structural splitting rather than sliding-window truncation by character count like ordinary documents; but code contains many cross-file dependencies, and pure text similarity cannot capture the calling relationships between functions—the call graph between functions is implicit structural knowledge that pure text retrieval inherently cannot perceive. This is precisely the core value of the "tracing calling relationships" step below—it upgrades static text retrieval into an awareness of the code's graph structure, enabling the system to follow the call chain to find code snippets that are semantically related but textually quite different.

The entire pipeline has five steps:

1. Split the code: Break code into meaningful small units—functions, classes, methods—rather than random chunks.
2. Run dual searches in parallel: Run semantic search and keyword search simultaneously, then merge the results. This is the primary source of token savings.
3. Compress further: Keep only function names and descriptions, compressing a 50-line function down to 5 lines.
4. Trace calling relationships: Record which function calls which, so that finding one snippet lets you follow the thread to related code.
5. Score and filter: Every result has a score; if the score is too low, it's not sent, preventing "bad context" from polluting the query.

Everything runs locally—nothing is uploaded to the cloud.

Why Run Dual Searches

Because each type of search has its own weaknesses, and these weaknesses are inherently determined by their underlying technical principles.

The Technical Complementarity of Semantic Search and Keyword Search: Keyword search (like the BM25 algorithm) is based on exact character matching—extremely fast but unable to understand synonyms and semantic variations. It only recognizes the exact letter sequence you write, and knows nothing about the semantic connection between "authenticate" and "login." Semantic search relies on vector embedding technology—converting code or text into points in a high-dimensional vector space (typically 512 to 4096 dimensions), where semantically similar content is closer together in the vector space, and relevance is calculated via methods like cosine similarity.

Common embedding models in the code domain include CodeBERT and GraphCodeBERT (the latter can also perceive the code's data flow graph), which are optimized specifically for code, as well as general-purpose models like OpenAI's text-embedding-ada-002. Vector retrieval is typically implemented with vector databases like FAISS, Chroma, and Weaviate—among which FAISS is Meta's open-source lightweight local solution, supporting efficient retrieval over billions of vectors, well suited for low-latency scenarios like the one in this article. The combination of both approaches is called Hybrid Search, and there's extensive research in the information retrieval field proving it outperforms either single method. RRF (Reciprocal Rank Fusion) is a common result-fusion algorithm that achieves normalized merging by summing the reciprocals of the rankings from both search paths; the weighted scoring formula the author adopted (50% semantic + 30% keyword + 20% recency) is a more direct and interpretable, pragmatic approach.

Semantic search excels at finding "meaning-related" code but misses exact names—if you search for authenticate user function, it might give you a different auth function with a similar meaning. Keyword search excels at exact names but misses related concepts—if you search for login flow, it will miss all code written as sign in.

Using either one alone, you miss roughly 1 out of every 4 results; combining the two drops the miss rate to about one in ten. They compensate for each other's weaknesses, jointly improving context quality.

The Hardest Part: Judging Relevance

The real difficulty isn't finding results—it's judging whether the results are actually relevant. Sometimes a search returns 10 results, and not one of them is right—if you feed these bad results to the AI, it will give a "confidently wrong answer," which is worse than no answer at all.

They tried having the AI judge the results itself—too slow, adding 2-3 seconds each time. They tried a fixed score threshold—too rigid; even a perfect match on a short question would be wrongly scored low.

Ultimately, they adopted a simple formula: 50% semantic score + 30% keyword score + 20% code recency, with the threshold dynamically adjusted based on the current results. The whole process takes only 0.4 milliseconds, with no extra AI calls needed. The core lesson: a simple formula beats a complex model in most scenarios.

This conclusion has theoretical support in the machine learning field, corresponding to the statistical extension of the famous "Occam's Razor" principle—there's a trade-off between model complexity and generalization ability (the Bias-Variance Tradeoff). In local code retrieval tasks with limited data and relatively fixed scenarios, a lightweight linear formula often has better stability than a deep neural network, is easier to debug and interpret, and has lower maintenance costs.

Real Data: Cutting Tokens by 94% Per Query

On a real open-source project (FastAPI, 53 files), they ran benchmarks using 20 questions that developers would realistically ask:

• Without the tool: 83,000 tokens per question
• With the local indexing tool: 4,900 tokens per question (a 94% reduction)
• With compression added: focused down to 523 tokens
• Accuracy: still finds the correct code 90% of the time

The testing methodology is fully public, and anyone can reproduce and verify it themselves.

Being Honest About the Limitations

Raj showed rare restraint about that 94% figure. This number is based on a "worst case"—reading all files every time. In reality, tools like Claude Code are already smarter than that, so actual savings will be lower than 94%.

An even more important limitation: large mixed codebases are very hard to handle. When testing on a large project with 396 files, recall dropped to nearly zero.

Recall and the Single Responsibility Principle in Code Architecture: In information retrieval, recall measures "of all the truly relevant results, how many did the system find"; precision measures "of the results the system returned, how many are truly relevant." The two often trade off against each other—loosening the retrieval threshold can improve recall but introduces noise, while raising the threshold lowers recall. This trade-off is called the Precision-Recall Tradeoff, a classic challenge in the information retrieval field.

In code retrieval scenarios, large mixed-responsibility files (a file containing many unrelated functions) severely interfere with the quality of vector embeddings, because the file's overall vector cannot accurately represent any single specific function within it. This causes all queries to hit this "semantically fuzzy" file while failing to find the specific logic actually needed. This also explains why codebases where "each file does only one thing" perform better—it corresponds exactly to the Single Responsibility Principle (SRP) in software engineering, systematically articulated by Robert C. Martin as part of the SOLID principles. This principle has gained new practical value in the AI era: good code architecture is not only friendly to human developers, but also directly improves the accuracy of AI retrieval tools, producing quantifiable economic benefits—a previously never-clearly-quantified causal chain has emerged between code quality and AI usage cost.

The pattern is: if each file does only one thing, it works great; if a file has mixed responsibilities, it fails. Additionally, for the sake of speed they chose a small, fast retrieval model, with index rebuilds taking less than a second—a large model would find more, but they chose speed over perfection.

The design philosophy of this solution has been consistent throughout: simple choices often beat complex solutions—a small database rather than large-scale infrastructure, two searches rather than one fancy algorithm, local rather than cloud.

Shared Index and Memory: Reusing Context Across Tools

Most developers use multiple tools simultaneously (Claude Code for hard problems, Cursor for quick edits, Copilot for small completions), but these tools each start from scratch every time and share nothing with each other—you have to explain the same codebase three times to three different AIs.

This problem has a fundamental technical cause: each AI tool maintains its own independent session state and context cache, and there's neither a communication protocol nor a shared storage mechanism between tools. As the number of tools grows, this waste of repeated explanation scales linearly.

The MCP Protocol and the Industry Trend Toward Standardizing Cross-Tool Context Sharing: This pain point is driving industry-level standardization efforts. The Model Context Protocol (MCP), launched by Anthropic in 2024, attempts to establish a unified context exchange standard, allowing different AI applications to share tool calls, resource access, and conversation history through standard interfaces. MCP adopts a client-server architecture, with the AI assistant as the client and external tools and data sources as servers, communicating via a standardized JSON-RPC message format, and supporting unified descriptions of three primitives: Resources, Tools, and Prompts. Similar to how USB unified hardware interfaces, MCP's goal is to end the fragmented status quo where "every AI tool has to individually adapt to every data source."

The shared index solution described in this article is highly consistent with MCP's design philosophy—both decouple context construction and management from the individual AI tools, forming an independent shared layer. This article simply chose to land the concept through engineering implementation before industry standards became widespread. As the MCP ecosystem matures, such local shared indexes are expected to be exposed directly to all compatible tools through standard protocols, at which point "index once, reuse across tools" will become an out-of-the-box capability rather than infrastructure you have to build yourself.

Their solution is to build one shared index, with all tools connecting to the same index and sharing the same set of search results. They also added a memory mechanism: when one tool learns something about the project, that knowledge persists to the next session, and switching to another tool lets you reuse the context directly, completely eliminating the waste of repeated explanation.

The scorecard on one real project: 247 queries, saving 12.4 million tokens, roughly $186 that went unspent. Of these savings, 84% came from the search layer, and the rest from compression. The tool tracks each query, compares "what would have been sent" against "what was actually sent," then multiplies by the model price to derive the real savings figure.

Conclusion: Optimizing Input Matters More Than Choosing a Model

Raj's core argument cuts straight to the pain point of AI coding today: we're always debating whether Opus or Sonnet is the best model, but model choice might only account for 30% of cost—the other 70% depends on what you feed it.

This judgment has a mathematical basis: at the same context quality, the price gap between different models is usually within 3-5x; whereas compressing context from 83,000 tokens down to 4,900 tokens amounts to nearly a 17x cost reduction. Compared side by side, the leverage from context optimization far exceeds that of model selection.

The answer isn't a more powerful model, but sending less, more precise context. This tool, called CCE, is already open-source and free—worth having every developer plagued by AI bills run it for a week to see their real numbers.

Key Takeaways

• The money is on the input side: Input context accounts for about 90% of total token cost; optimizing output is just fiddling with the small numbers.
• RAG is the essence of the solution: The core of the local code search layer is the retrieval-augmented generation architecture, turning "send everything" into "retrieve on demand."
• Hybrid search beats single methods: Semantic search and keyword search each have blind spots; combining the two can drop the miss rate from 25% to about 10%.
• Code architecture affects AI cost: The Single Responsibility Principle isn't just an engineering norm—it directly determines the accuracy and economic benefit of AI retrieval.
• A simple formula beats a complex model: A 0.4-millisecond weighted scoring is far superior to a 2-3 second AI self-assessment; in constrained scenarios, Occam's Razor holds.
• A shared context layer is the trend: Industry protocols like MCP are standardizing this capability, and landing it via engineering ahead of time is a pragmatic choice.