Book the Partial Before You Retry: Abandoned Free Model Calls Aren't Free

Title

Book the Partial Before You Retry: Abandoned Free Model Calls Aren't Free

Meta Description

Abandoned free model calls still burn queue time and emitted tokens. This article provides a local ledger script that decides whether to retry the free lane, cut over to paid, or stop — by tracking elapsed time, spent tokens, and self-imposed ceilings.

Article Content

Free model calls are not free. When you cancel a streaming request, you have already consumed queue time and the tokens the server has already produced. If you simply retry, you are effectively running a loop that looks frugal but is billed by the minute. This article introduces a retry decision method based on a local ledger.

The Core Problem: An Abandoned Call Is Not a Sunk Cost

When you hit slow queues on a free shared server, clients tend to be overly optimistic. The first attempt streamed back a few hundred tokens, but the time window you promised the user is about to expire, so you cancel and resend the same prompt. The second attempt queues behind the first because closing your socket doesn't pull the job out of the pool. The zero price on the token billing sheet doesn't roll back that queue.

This is a ledger problem, not an HTTP client's mood problem.

The reason the "queue" doesn't disappear just because the client closes the socket is that free shared inference services typically use request queueing with batching/scheduling systems: your HTTP connection is only the pipe carrying results back, while the job itself has already entered the backend scheduler's waiting queue or an executing batch as a separate task. Cancelling the connection generally only stops streaming tokens to the client — it does not automatically cancel a job that's already queued ahead or already running on the GPU. Many implementations don't even expose a "cancel job" interface. So the queue position the first attempt occupied, the compute it was allocated, and the tokens it generated all continue to bill to the server's total ledger (even if they show as $0 to you). This also explains why the immediately following second request queues behind the first: the scheduler sees two independent jobs, not a retry of the same job. Understanding this is the prerequisite for turning "cancel and retry" from a psychological comfort into an engineering decision that requires accounting.

Three Numbers You Need to Track

Update three numbers after each attempt:

• spent_s: queue time plus generation time you have already waited
• spent_tokens: the number of tokens already returned, including parts you won't deliver
• ceiling_tokens: the token ceiling you set for this job, not a vendor quota copied from some article

The ceiling exists so the free lane doesn't become an unbounded loop inside your own process. A retry is only allowed if the next attempt can complete within the remaining deadline, and if spent tokens plus a new worst-case estimate remains under the ceiling. Otherwise, cut over to the paid lane for one clean attempt, or stop and return what you have.

A Runnable Ledger Script

The following script book_partial.py is a local decision aid — it is not a client for any vendor, and it does not open sockets.

#!/usr/bin/env python3
"""Write off partial free-lane attempts before you allow another retry."""

from __future__ import annotations
import json
import sys
from dataclasses import dataclass

@dataclass(frozen=True)
class Attempt:
    queue_s: float
    gen_s: float
    emitted_tokens: int
    shipped: bool

@dataclass(frozen=True)
class Policy:
    deadline_s: float
    ceiling_tokens: int
    next_queue_s: float
    next_gen_s: float
    next_est_tokens: int
    paid_queue_s: float
    paid_usd_per_1k: float

def book(attempts: list[Attempt], policy: Policy) -> dict:
    spent_s = sum(a.queue_s + a.gen_s for a in attempts)
    spent_tokens = sum(a.emitted_tokens for a in attempts)
    remaining_s = policy.deadline_s - spent_s
    next_free_s = policy.next_queue_s + policy.next_gen_s
    next_paid_s = policy.paid_queue_s + policy.next_gen_s
    projected = spent_tokens + policy.next_est_tokens

    if any(a.shipped for a in attempts):
        action, why = "stop", "a previous attempt already shipped"
    elif projected > policy.ceiling_tokens:
        action, why = "stop", "another try would cross your token ceiling"
    elif next_free_s <= remaining_s:
        action, why = "retry_free", "remaining clock covers one more free attempt"
    elif next_paid_s <= remaining_s:
        action, why = "cut_over", "free retry does not fit; one paid attempt does"
    else:
        action, why = "stop", "no lane finishes inside the time you have left"

    sunk_usd = (spent_tokens / 1000.0) * policy.paid_usd_per_1k
    return {
        "action": action,
        "why": why,
        "spent_s": round(spent_s, 2),
        "remaining_s": round(remaining_s, 2),
        "spent_tokens": spent_tokens,
        "projected_tokens": projected,
        "sunk_usd_if_those_tokens_were_paid": round(sunk_usd, 4),
    }

def main() -> int:
    raw = json.load(sys.stdin)
    attempts = [Attempt(**item) for item in raw["attempts"]]
    policy = Policy(**raw["policy"])
    json.dump(book(attempts, policy), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The script is deliberately written as pure functions plus data classes rather than an HTTP client wrapper. The reason: separating decision logic from network IO lets you repeatedly verify branch behavior locally with fixed inputs, unaffected by network jitter and real quotas. Attempt records the physical cost of one attempt. Policy records the constraints for this job (deadline, token ceiling) and estimates for future lanes (queue time, generation time, and expected tokens for the next free/paid attempt). book() takes only these two inputs and outputs one action plus one reason. So it can be called synchronously as a "budget checker" before real calls, or replayed from logs for historical decisions. Putting estimates in Policy rather than hardcoding them in the function lets offline sampling results (see next section) be injected via configuration, avoiding code changes for every adjustment.

The Three Decision Branches in Practice

The first test case is an overnight job with a generous time window. One partial return, nothing shipped, still room under the self-imposed ceiling.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

The result should be retry_free. 44 seconds spent, ample remaining window, and 220 plus the estimated 800 is still below 4000.

The second is an interactive scenario: same partial return, but a 60-second deadline.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 60,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

16 seconds remain. The next free attempt needs 47 seconds, while one paid attempt needs 14 seconds — still feasible — so the action is cut_over.

The third case drops the ceiling to 500, even with a generous deadline:

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 500,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Projected tokens are 1020, exceeding the 500 ceiling — so stop.

Pin Branch Behavior with Assertions

To prevent the three branches from drifting when you edit formulas, add a local test:

python3 - <<'PY'
import importlib.util
spec = importlib.util.spec_from_file_location("book_partial", "book_partial.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)

def run(deadline, ceiling, emitted=220):
    attempts = [mod.Attempt(35, 9, emitted, False)]
    policy = mod.Policy(deadline, ceiling, 35, 12, 800, 2, 0.002)
    return mod.book(attempts, policy)["action"]

assert run(600, 4000) == "retry_free"
assert run(60, 4000) == "cut_over"
assert run(600, 500) == "stop"
assert run(60, 500) == "stop"
print("ok")
PY

Note: the ceiling is checked before the deadline, so when both trigger, stop takes precedence over cut over.

How to Sample Queue Time

Before trusting next_queue_s, sample during the hours you actually run. Send a short fixed prompt to the free lane, record the time-to-first-token as queue time and the rest as generation time. Do this three times, take the worst queue time, and discard the probe text. If the probe fails, the free lane is unavailable for that hour.

Don't repeatedly retry the probe until it feels okay — that loop is exactly what the ledger exists to prevent.

The emphasis on "sampling during the hours you actually run and taking the worst queue time" is because free shared queue latency typically shows extreme variation by time of day: a few seconds at night and early morning, but tens of minutes during daytime peaks before your job gets its turn. Filling next_queue_s with an average or best-case value makes the ledger systematically underestimate cost, leading to frequent wrong decisions of "try the free lane once more" and blowing the response window the user can tolerate. Taking the worst sample is a conservative bias — the cost is possibly cutting over to the paid lane too early, but the payoff is predictability of the deadline. The essence of these rules — sample three times, take the worst, discard probe text, and treat probe failure as free-lane-unavailable for that period — is: better to admit "the free lane is unavailable right now" than to repeatedly gamble for luck just to save one paid call.

Where the Free Lane Doesn't Belong

Free capacity is the wrong choice when someone is blocked waiting for an answer, when the attempt you're abandoning is not idempotent, or when you cannot sample queue time. It's also wrong when a retry could apply side effects twice. Stamp the job with your own idempotency key before the first call. Treat shipped: true as a final state, so a late free response doesn't overwrite a paid answer already returned.

What This Article Does Not Cover

The function trusts your sampling. A quiet afternoon doesn't represent a week — if the queue is volatile, keep the worst sample from your running hours and fill in next_queue_s. It treats all emitted tokens as spent, which is the correct bias for the ceiling and the wrong bias for quality scoring.

If your client supports continuation, just add the missing gap to the projection and rerun the assertions. Don't use this for payments, access control, or any path where stopping is unacceptable.

One-line Summary

Free model calls aren't free: use a local ledger to track spent queue time and tokens, then decide whether to retry, cut over to paid, or stop.

Paragraph Summary

The article points out that cancelling a streaming free model call does not remove it from the server queue — the spent queue time and emitted tokens still bill to your own engineering ledger. So "cancel and retry" is effectively running a loop that looks zero-cost but consumes time. The author argues for modeling this as a ledger problem: update spent_s, spent_tokens, and a self-imposed ceiling_tokens after each attempt, then decide across three branches — "retry free," "cut over to paid," or "stop and return what you have." To that end, it provides a pure-local Python script book_partial.py that opens no sockets, demonstrates the three branches with three test cases, and shows how to pin behavior with assertions. It also stresses sampling actual queue times and using the worst sample as the assumption for the next free attempt, and warns that the free lane is unsuitable when someone is blocked waiting, when operations aren't idempotent, or when double side effects are possible.