Your Repo Doesn't Fit in the Context Window? Measure Tokens Before You Design

After two days of development, the author added a piece of code to the cartographer AI agent that "concatenates the entire repository into a single string." He ran len() on it out of habit—and then discovered the problem: he had no idea how many tokens that text corresponded to, nor how much a single run would cost.

So the task for day three was: don't write new features, build a ruler and an electricity meter first. Behind this decision lies a constraint that every agent developer will eventually face—the repo will always be larger than the context window.

Repos Are Measured in Characters, Models Charge by Token

Repository size on disk is measured in characters, while the unit models read and write in is tokens. The conversion ratio between the two is not a constant you can look up.

The author provides a repository traversal function source_files() that walks all files under the root directory, keeps only those whose extensions are in SOURCE_SUFFIXES (Python, docs, config), and excludes files whose paths contain any of SKIP_DIRECTORIES. measure() returns the file count and total character count, ignoring illegal text during reading.

It's worth emphasizing the directory-skipping layer. If you don't filter out .venv and node_modules, you're not measuring the project itself—you're measuring its dependencies, which is a much larger number with no reference value for code size. Measure the code humans wrote, not everything installed alongside it.

Token is the basic unit by which large language models process text. Models don't read characters directly; instead, a component called a tokenizer first splits text into segments from a vocabulary: common English words often correspond to a single token, while long identifiers, rare words, punctuation, newlines, and indentation may be split into multiple. As a result, the same source code can produce different token counts across different models, because each model family was trained with its own vocabulary and segmentation rules. This is why there's no universal conversion ratio between character count and token count—it's model-dependent and drifts with language and coding style. Context windows, rate limits, and billing are all measured in tokens, which is why the intuitive question "how big is my repo" must be translated into "how many tokens will it cost to call this model" to have any engineering meaning.

Two Wrong Ways to Estimate

Character count isn't a bill, so you need a ratio. The two common approaches are both wrong:

• characters // 4—a folk constant that corresponds to no specific model and suits source code poorly. Indentation, punctuation, and long underscore-separated identifiers in source code all make it diverge significantly from tokenization results on natural language.
• Using another vendor's tokenizer library for exact computation—this looks more rigorous but is actually worse. Your model uses its own vocabulary; an "exact number" computed with another family's tokenizer is for a vocabulary that nobody in your request is even using. Precision is not accuracy, the bias is still systematic, and it's unobservable from the outside.

The author's judgment is blunt: better imprecise than wrongly precise.

Vocabulary is the lookup table a tokenizer uses to map a string into integer IDs, typically trained statistically on large corpora by algorithms like BPE (byte pair encoding). Different vendors and model generations have independent vocabularies, so "using vendor A's tokenizer to precisely compute vendor B's model's token count" is logically incoherent: the number produced is for a vocabulary that will never be used in your request. This error isn't random noise—it's a stable bias in one direction, and it's unobservable from the caller's perspective. To get authoritative numbers, you can only rely on usage.prompt_tokens returned by the server, which is counted after the model's actual tokenization.

Two Samples + One Subtraction = Real Ratio

The truly authoritative counter comes from the server: the usage.prompt_tokens in each response is the input token count the server actually counted, at almost zero cost.

The author's prompt_tokens() function sends a request with max_tokens=1 just to get this field—because it's measuring input, not generating an answer.

But don't send just one sample and divide. prompt_tokens counts the full rendered request, including the fixed overhead of the chat template. When the sample is short, the fixed overhead dominates, and dividing it in permanently contaminates the ratio.

The correct approach is to send two samples with clearly different sizes and take the slope:

def chars_per_token(count, small, large):
    return (len(large) - len(small)) / (count(large) - count(small))

The constant term cancels out in the subtraction. Two requests and two output tokens give you a ratio measured for the exact model you're calling. Change models and it no longer applies—which is precisely why it should be a function rather than a constant in a config file.

The accompanying offline test is also crucial: given a fake counter with known fixed overhead of 11 tokens and a true ratio of 4, assert that the result is exactly 4.0. If the subtraction logic is wrong, this assertion fails.

Input and Output Must Be Accounted Separately

The other half of why the meter exists is that input and output tokens are not interchangeable. The author's Meter class accumulates prompt_tokens and completion_tokens separately and provides a dollars property converted from per-million-token prices.

Three details are worth noting:

1. All prices come from environment variables, defaulting to 0. No price literal is written into the source code—published prices change, and a stale number in version control is worse than no number.
2. On free models, dollars is honestly 0, but token counts remain real. Because what runs out first is never money but tokens: the window fills up first, rate limits hit first.
3. Separate accounting also tells you which lever to pull. Conversation history is resent every turn, so input keeps growing; output is usually just a paragraph. The optimization methods for each side are completely different.

record() is called by post() on every response, ensuring no request can bypass the meter. The test file test_budget.py locks down three things offline: ratio calibration, the skip list, and pricing logic.

Making the Repo Fit: Three Paths

	Send whole repo	Index everything upfront	Agent picks what to read
Cost	Pay for the whole repo on every call	One-time embedding (turning each file into a searchable number), then cheap queries	Pay only for files actually opened
When it fails	Repo exceeds the window—soon	Index is stale after the next commit	Agent picks wrong
Best for	One question, one small repo	Stable code, many repeated queries	Large repos, open-ended questions

The author explicitly chooses the third column, and not reluctantly. A map generated from all content is not a map—that's the territory itself. Choosing what matters is the real product; the constraint merely forces the design to be honest about this.

This aligns with the mainstream thinking in current AI coding tools—as multiple developers have repeatedly noted in public discussions, the ability to retrieve, select, and trim context is shifting from a "supporting feature" to the product's competitive edge itself. The author also put measurement results pinned to a commit SHA in docs/measurements.md rather than hardcoding them in this article: measuring against a moving branch isn't measurement, and freezing numbers into a blog post is choosing the wrong place entirely.

Embedding maps a piece of text into a high-dimensional vector so that semantically similar texts are closer together in vector space. An embedding-based retrieval pipeline typically works like this: pre-generate vectors for each file or code block and store them in an index; at query time, convert the question into a vector too, and take the most similar blocks into context. Its value lies in turning "stuff the whole repo in" into "fetch a small relevant piece," thereby capping per-call cost at a constant level. The cost is that the index must be rebuilt as code changes, and recall quality depends on chunking and vector quality. Understanding this helps clarify the failure mode of the second column in the table: once the index is stale, retrieval results silently drift from the real code, and the caller often doesn't notice.

Five Pieces of Advice for Agent Developers

1. Measure before you design. A few seconds of len() can tell you whether the architecture in your head is even legal.
2. Calibrate against the model you actually call. Two samples, one subtraction—don't use folk constants or another vendor's tokenizer.
3. Keep prices out of source code. Use environment variables, defaulting to 0.
4. Print the meter on every run, including free-tier runs. Forming the habit is itself the goal.
5. Treat "selection" as a feature. Once you admit the agent must decide what to read on its own, the tool shifts from convenience to product.

Day three produced no new features, but the project now has units of measurement. For a project destined to hit the context window, this may be the most worthwhile day of all.

One-line Summary

Build the agent a token ruler and an electricity meter before discussing how to fit a repo that won't fit.

Paragraph Summary

This article documents the third day of an agent project: no new features, but establishing measurement first. The core constraint is that the repo will always be larger than the context window, and there's no lookup-able conversion constant between disk character count and the token count models charge by. The author advocates calling the server with two differently-sized samples and taking the slope of usage.prompt_tokens to calibrate the ratio, eliminating the chat template's fixed overhead, while rejecting folk constants and other vendors' tokenizers. A Meter class separately tracks input and output tokens, with prices from environment variables. Facing an oversized repo, the author clearly chooses the third of three paths—sending the whole repo, indexing upfront, or the agent selecting what to read—arguing that "choosing what matters" is itself the product.