Routing Models with a Token Ledger: A 60-Line Python Approach to Cost Control

Runaway costs in multi-model setups usually aren't caused by picking the wrong model — they happen because the decision to call has never been recorded. A classic failure looks like this: a script fires one request per changed file, the longest file pushes the prompt past the cheap endpoint's limit, the retry loop kicks in as usual, and a single code review turns into a billing event. Nobody chose this; it simply fell out of a for loop that never asked "how big is the thing I'm about to send?"

The original author's fix is refreshingly plain: spend two days not tuning prompts, but instrumenting the decision itself. The question shifts from "which model is better" to "which call is allowed to happen," and the latter can be answered with roughly 60 lines of Python and a single JSONL file.

Record First, Route Later

Most routing advice starts with model quality, but for personal projects that order is backwards — quality is the thing you can't measure cheaply, while call volume is the thing you can. The author's stance: write down every call first, then argue about where it should go.

The estimator is deliberately crude: deterministic, no network round-trip, pessimistic about code text. The core is the empirical value CHARS_PER_TOKEN = 3.6, aimed at prompts mixing prose and code. The value of this choice is reproducibility — anyone running the same text gets the same number, with no model-side randomness.

Two design details in the ledger records are worth borrowing. First, the prompt is hashed with SHA256 and truncated before storage, which lets you find duplicate calls without retaining the original prompt — the author specifically notes that the diffs used for review aren't data he's allowed to keep. Second, wall_s records measured wall-clock time rather than an estimate, so the free path's latency can be judged on real numbers instead of vibes.

How CHARS_PER_TOKEN is chosen and where the error comes from

CHARS_PER_TOKEN = 3.6 is a rough conversion heuristic for English prose and common source code: OpenAI's BPE tokenizer averages about 4 characters/token for English, but the abundance of symbols, indentation, and short identifiers in code drags the ratio down to roughly 3.5–3.8. Choosing a smaller value means the estimated token count comes out higher — that is, more pessimistic about cost, which is exactly the direction the designer deliberately chose.

The error grows significantly on several kinds of text: minified JSON or JS contains almost no whitespace, so the characters/token ratio can drop to 2.5; non-Latin scripts such as Chinese, Japanese, and Korean often cost 1–2 tokens per character. So this number must not be used to reconcile against a bill — only to rank calls within the same script relative to each other. Comparing "is this one bigger than last time" is reliable; asserting "this will cost X" is not.

The Routing Table Is the Whole Design

There's nothing clever about the routing rule itself: a lookup based on two booleans and a size. The value is precisely that the rule is written down somewhere a reviewer can object to it.

Condition	Route	Rationale
Requires strict JSON or a pinned model version	pinned	schema drift is worse than cost
Estimated tokens exceed the free path's limit	pinned	truncating mid-diff silently produces garbage
Everything else	free	this is volume traffic, and volume is exactly what free access is for
Retry after failure	pinned	a retry is not an experiment; pay for determinism

The last row is the one the author is most willing to defend: retrying with a different sample isn't a retry, it's a second experiment — and experiments should be labeled as experiments.

This table also explains why "record first, route later" is necessary. Without a ledger, you can't know where the free path's limit should be set, nor which calls are actually duplicates.

What pinned and free endpoints mean in engineering terms

Pinned means the request is firmly bound to a specific model version or provider endpoint, without accepting server-side automatic fallback, sampling-parameter drift, or version rolling; free means relying on free quota or free-tier access, typically carrying uncertainty such as rate limits, quota caps, instance hibernation, or model versions changing at any time. The difference isn't just price, it's predictability: the pinned path produces reproducible output and trustworthy schemas, while the free path suits volume calls that don't care if results fluctuate. The reason the routing table can be expressed with two booleans and one threshold is that it breaks the single question "can this call tolerate uncertainty?" into three decidable conditions: structural constraints, size constraints, and failure handling.

What Two Days of Traffic Looks Like

An instrument nobody reads is just disk usage. Each night the author runs a summary script that outputs the distribution grouped by route: median tokens, p95 tokens, median latency per group, plus a count of duplicate prompts derived from the hashes.

The duplicate count is a number the author didn't expect to care about. Duplicate prompts are pure waste at any tier, free or not, and before you hash prompts these duplicates are completely invisible.

Worth noting is the author's restraint on data disclosure: he explicitly declines to publish the percentage saved over his two-day window, on the grounds that "one repo on one weekend is not evidence — citing it is citing a benchmark nobody can reproduce." His position: the script is the argument; run it on your own traffic and you'll have numbers in a night. That attitude is rare in AI content that routinely claims "80% cost savings," and it's also more credible.

Running the Routing Service on a Small Free Instance

The routing service itself does almost nothing — one file, one port, no database — which makes it a reasonable candidate for a small instance rather than a paid one. The author mentions MonkeyCode's free model access and free server options, and the workflow depends on both: the free path handles volume calls, and the free instance hosts the ledger service.

To be clear, this is an article with a promotional angle, and the author discloses that. His approach: treat free quota and free servers as "availability claims from the operator," trusting them only after verifying in his own environment. Specifically, run the smoke test on day 0 rather than day 30 — hit the routing endpoint with 20 concurrent requests, tally HTTP status codes and response times, then use systemd with Restart=on-failure as the sole resilience promise, paired with journalctl -f to watch behavior under burst traffic. If the free instance hibernates, you'll see it in wall_s first, not on the bill.

Smoke tests and systemd's Restart=on-failure

A smoke test means verifying key paths with a small number of real requests before a system goes live, confirming basic availability; here it concretely means hitting the routing endpoint with 20 concurrent requests and tallying HTTP status codes and response times, so that rate limiting or cold starts on the free instance surface on deployment day rather than a month into operation. Restart=on-failure is a systemd service unit setting meaning the process is automatically restarted by systemd when it exits with a non-zero status or is killed by a signal; it does not handle hangs or false liveness, so it must be paired with journalctl -f to track logs in real time and observe actual behavior under burst traffic. Together they form a "minimum-cost resilience promise": not a guarantee that the service is always up, only that it's automatically relaunched after a crash and that the process is observable.

Pitfalls and What's Worth Reusing

The bad news was predictable: the estimator over-counted long diffs badly enough to push medium-sized prompts onto the pinned path, and the initial limit was set too low. Eyeballing a higher number isn't the fix — the real fix is reading the free route's tokens_median and p95 and setting the limit at the 95th percentile of what you actually send.

What's worth reusing is the order: ledger first, routing second, deployment last. Do it backwards and you'll end up with an unauditable routing policy and a server you're afraid to restart.

The author also candidly notes two limitations. Character-count token estimates are wildly off on minified files and non-Latin text, so don't use them for budgeting — only for relative comparisons between calls. And the design says nothing about answer quality — it decides where a call goes, not whether the answer is good.

As for who should skip all this: people with latency SLAs, people whose diffs can't leave the intranet, and people whose traffic is so small that the ledger costs more attention than the calls themselves. If you fall into any of those categories, hardcoding a single endpoint is the correct engineering choice.

The persuasiveness of this approach isn't in the line count — it's that it breaks a vague cost anxiety into three steps: recordable, look-up-able, auditable. For any solo developer manually switching between endpoints, build the ledger first, and the remaining decisions will clarify themselves.

Why use p95 rather than the mean to set thresholds

p95 is the value at the 95th percentile after sorting all observations, meaning 95% of calls fall at or below it. Compared with the mean, p95 isn't dragged down by a few tiny values nor dominated by extreme outliers, so it reflects more truthfully "how big a prompt gets sent in the vast majority of cases." The author's initial lesson was exactly that the threshold was set too low, causing the estimator's over-counting of long diffs to push medium-sized prompts onto the pinned path; switching to the tokens_median and p95 of actual send sizes to locate the limit amounts to letting observation, not gut feeling, decide the threshold. The same applies to latency metrics: the free path's wall_s must be judged by p95 rather than the mean to reveal the long tail caused by hibernation or rate limiting.

One-line Summary

Record the tokens and route of every call first, then decide where calls go with a lookup rule.

Paragraph Summary

The article argues that multi-model cost control should record before it routes: use a ~60-line Python script with a deterministic estimator at CHARS_PER_TOKEN=3.6 to size each prompt, write each call's hash, estimated tokens, route choice, and measured wall-clock time into a JSONL ledger, then use a lookup table with three conditions — does it need strict JSON, does it exceed the free path's limit, is it a retry — to decide between pinned and free endpoints. Each night the ledger summarizes median and p95 tokens, latency, and duplicate prompt counts per group. The author stresses that estimates are only for relative comparison between calls, not absolute budgeting; retries must go to a pinned model for determinism; and a small free instance hosts the routing service, with smoke tests and systemd auto-restart as the minimum resilience guarantee.