Before You Ship AI Features, Build a Spend Breaker

Title

Before You Ship AI Features, Build a Spend Breaker

Meta Description

Essential reading for indie developers before shipping AI features: build a Token spend breaker with three pieces—ledger, gate, and settlement—to prevent a Worker retry loop from eating your free quota over the weekend. Includes complete Node.js code and offline tests.

Article Content

An indie developer shipped an AI button on a Friday afternoon that turned messy customer messages into three bullet points. The demo looked good, the weekend started as usual, and nobody was watching that endpoint.

By Monday morning, that endpoint had handled tens of thousands of calls. A Worker's retry loop kept hammering away, stuck on a timeout. The free quota was gone, and every subsequent request was burning real money.

The numbers in the story are illustrative, but the failure structure is not. Free tiers never eliminate a budget—they just change the unit from dollars to Tokens and CPU. When an AI feature goes off the rails at 3 a.m., who's paying? For indie developers, the answer has to be written into the code, not left to human attention.

The Loop That Eats Your Free Quota

Most first versions look like this:

// worker.mjs
async function summarize(note) {
  while (true) {
    try {
      return await callModel(note);
    } catch {
      // retry forever, because "free"
    }
  }
}

This loop has no brakes. A single timeout, a 429, or a slow backend can turn one user action into thousands of billed calls. Providers are happy to keep answering until the quota is exhausted, then either charge or refuse.

The open-source project MonkeyCode offers free model access and a free server option, both provided by the operator. The free Token allowance is currently listed at 10 million. Check the project's latest terms before relying on any specific number, since quotas and limits change. (Disclosure: this article is part of a MonkeyCode product promotion.)

A free quota is still a budget—it's just spent in Tokens rather than dollars, and it refreshes on someone else's schedule. That's why the breaker counts Tokens first, and money second. On day one of any indie project, that order matters.

Exponential Backoff and Jitter

The while (true) in the example is dangerous because it implicitly assumes zero delay and unlimited retries. In production, the standard response to a 429 or 5xx is exponential backoff: wait about 1 second on the first retry, then grow by powers of 2 each time, with a maximum retry count. Plain backoff still creates a "thundering herd"—many clients retrying in lockstep at the same moment, amplifying a momentary failure into sustained overload. So layering random jitter on top of the backoff interval, so each wait falls within a range like [base, 2×base], is a common combination.

For the breaker, backoff solves "don't overwhelm the provider yourself," while bounded retries solve "don't let one user action become thousands of billed calls." Both must be present: backoff without a cap still lets Token consumption accumulate linearly during a long outage; a cap without jitter can produce periodic billing spikes across multiple Workers. The article's recommendation to "retry twice, then degrade" is exactly putting those two constraints into concrete numbers.

Three Pieces: Ledger, Gate, Settlement

The breaker is small and deliberately boring. An append-only ledger, a pre-call gate, and a post-call settlement.

// spend.mjs
const TOKEN_BUDGET = Number(process.env.TOKEN_BUDGET ?? "200000");

export function estimateTokens(text) {
  // rough heuristic: about 4 characters per Token for English prose
  return Math.ceil(String(text).length / 4);
}

export function gate(text, now = new Date()) {
  const estIn = estimateTokens(text);
  const s = spendToday(now);
  const projected = s.in + s.out + estIn + EST_OUT;
  if (projected >= TOKEN_BUDGET * 0.9) {
    return { ok: false, reason: "token-budget", estIn, ...s };
  }
  return { ok: true, estIn, ...s };
}

The gate closes when 90% of the budget is used, not at 100%. Once it's closed, you still have to answer the user, so the wrapper returns a deterministic fallback instead of an error page.

export async function guardedSummarize(note, { callModel, now }) {
  const verdict = gate(note, now);
  if (!verdict.ok) {
    settle({ route: "fallback", in: 0, out: 0, usd: 0, reason: verdict.reason }, now);
    return { text: cheapFallback(note), route: "fallback" };
  }
  try {
    const res = await callModel(note);
    settle({ route: "model", in: res.usage?.input_tokens ?? verdict.estIn, ... }, now);
    return { text: res.text, route: "model" };
  } catch (err) {
    settle({ route: "error", in: verdict.estIn, out: 0, usd: 0, error: String(err) }, now);
    throw err;
  }
}

Note the catch branch: failed attempts usually still consume input Tokens, so you must record them even when the request errors. Otherwise your book spend will undercount the real bill, and the breaker will only close at the wrong moment.

The model call itself lives behind a thin adapter. Any OpenAI-compatible endpoint works, and the breaker won't lock an indie project into a single vendor. baseUrl, apiKey, and model are read from environment variables—never hardcode secrets into a repo that gets cloned by a free host.

Append-Only Ledger

The ledger is a write-only, never-edited-in-place sequence of events: each call appends a row containing timestamp, route, input/output Token counts, estimated or actual dollar cost, and reason. To correct a mistake, you can't rewrite an old row; you append a reversing entry. This structure brings three benefits: you can fully replay what happened that day during an audit; there's no "partial write" intermediate state after a crash (the worst case is losing the last row); and you can aggregate spendToday(now) on the read side by day.

The cost is that the file keeps growing. For a single-process indie app, sharding by day (e.g., spend-2026-10-16.jsonl) is enough—both easy to archive and easy to delete. Since the ledger is stored as plain-text JSONL, it doubles as a cheap debugging tool: when the gate unexpectedly closes, searching that day's route: "error" records directly is often faster at locating the runaway retry source than digging through logs.

Prove the Gate Closes with a Test

A breaker that's only been read and never executed is worthless. Here's a fully offline test using a stub model and an explicit clock:

test("the gate closes before the budget is spent", async () => {
  const now = new Date("2026-10-16T09:00:00Z");
  const routes = [];
  for (let i = 0; i < 40; i++) {
    const out = await guardedSummarize(note, { callModel, now });
    routes.push(out.route);
  }
  const total = spendToday(now);
  assert.ok(total.in + total.out < 1000, "stayed under the hard token budget");
  assert.equal(routes.at(-1), "fallback", "the gate closes and stays closed");
});

The first test loops forty times and still finishes in milliseconds. That's exactly the point of an offline stub. The failure you're worried about once cost someone a weekend; here it costs one test run.

Injectable Clock and Deterministic Testing

Code that depends on real Date.now() is hard to test for cross-day behavior: you can't verify "a new day resets the budget" without waiting twenty-four hours. Passing now as a parameter and explicitly constructing time points in tests is the standard trick for turning time into a normal input. Production code uses new Date() as the default, while tests pass in a fixed moment, so the same logic works against real time in production and can walk through arbitrary time sequences in milliseconds in tests.

The stub model works the same way: replace callModel with a fake function that returns a fixed usage, and the test needs no network, no API Key, and produces no cost. That's what lets budget-related assertions like "the gate closes" enter regular CI—because a failing budget test that accidentally calls a remote model would itself cause damage. The article's emphasis on "finishing in milliseconds" lands exactly on this kind of repeatable, zero-cost, zero-spillover verification.

Where the Free Server Fits

Run the Worker and the ledger in the same process on the same small host. The free server option fits exactly this shape: one Node process, one SQLite or JSONL file, one external model endpoint.

Put the ledger path inside the app directory, not /tmp. A restart shouldn't silently refund the day's spending. Also constrain the process to a single instance, since two Workers writing the same file can cause double-counting or dropped rows. Free hosts have their own limits; read the current terms clearly before promising anything to customers, and decide how the degradation path behaves on cold starts or when rate-limited.

The First Month's Routing Matrix

Scenario	Route	Why
User clicks, waiting	Model call with gate	Latency acceptable, one call per action
Background task iterating over many rows	Batch with a per-run cap	Runaway loops start in Workers, not the UI
Gate already closed	Deterministic fallback	Deliver a worse answer, not an outage
Provider returns 429 or 5xx	Retry twice, then degrade	Unbounded retries multiply Token spend
Accounting must be exact	Neither	Use a billing system, not a JSONL file

Limitations and Who Should Skip This

The ledger is in-process and file-based. It suits a single indie app, not cross-region deployments. The 90% threshold is just a guess and should be tuned to real traffic. Token estimates deviate from provider counts; when the endpoint returns a usage report, settle against the report.

More importantly, the breaker protects quota, not correctness. A bad prompt still returns bad summaries within budget. Free model access, like any free tier, is a moving target—verify the current terms before building business rules on top of it.

If you already have provider-side quotas and a billing dashboard, you can skip this. For high-throughput services, put a gateway in front of every call—also skip this. If you can't re-read the provider's limits repeatedly, skip it too. One indie project plus one AI button is the sweet spot, because one JSONL file and forty lines of Node code buy one guarantee that really matters: the weekend stays boring.

The Gateway Approach for High-Throughput Scenarios

When call volume scales to multiple instances and multiple regions, an in-process JSONL file hits two problems at once: concurrent-write races (two processes read-modify-write the same counter, causing under- or double-counting), and an inability to share state across geographies (a Worker in Singapore can't see spend in Frankfurt). At that point you typically introduce a standalone gateway service, routing all model requests through it, where the gateway centrally checks quotas before calls and centrally writes to shared storage (Redis, a database, or provider-side metering) after calls.

The cost of a gateway is one extra hop of latency and an additional operational surface. For an indie project with one AI button, that overhead far outweighs the benefit; but once daily calls reach the millions, an in-process breaker degenerates into "each instance doing its own thing," unable to honor a global budget promise. The criterion isn't technical sophistication—it's whether you need a cross-process single source of truth.

One-line Summary

Free AI quotas are still a budget—compute Tokens before money in a code-level breaker.

Paragraph Summary

The article opens with a fictional story of an indie developer burning through a free quota over a weekend, pointing out that free tiers don't eliminate a budget—they just change the unit from dollars to Tokens. The author proposes a deliberately boring breaker to prevent runaway spend: an append-only ledger handles accounting, a pre-call gate closes at 90% of budget and returns a deterministic fallback, and a post-call settlement records input Tokens consumed even when requests fail. Model calls sit behind a thin adapter to avoid locking into a single vendor. Using offline stub models and an injectable clock, tests can verify within milliseconds that the gate really closes. The approach suits single-process indie apps, not multi-region deployments or high-throughput services, and the free quota itself is a constantly shifting moving target.