Stream Interruptions Aren't Free Retries: Save Repeated Token Costs with Cursor-Based Resumption

When an LLM streaming request gets preempted, most people's first instinct is to resend the entire prompt and hope the next available slot is more cooperative. This seemingly harmless reflex turns a brief interruption into paying twice for the same answer. This article outlines a lightweight operational habit: continuously tracking a prefix cursor during streaming output so that, after an interruption, you only resume from where you left off — not from the beginning.

It's worth noting upfront: the original author explicitly states this is a proposal, not a tested result. The example code has not been run in production, makes no assumptions about specific model names, quotas, or machine specs, and makes no promises about whether any free tier will still exist next week.

Why Resending the Entire Prompt Means Paying Twice

The core insight is: a preempted stream is not a free retry. If the channel has already emitted tokens, those tokens have already been consumed. Sending the same message again is like buying a second copy of a prefix you already hold.

The original author uses a fitting analogy: a delivery driver has already brought your package to your doorstep, then gets rerouted. You wouldn't hire a second driver to travel the full route from the warehouse — you keep what was already delivered and only pay for the last flight of stairs. Streaming model output has exactly this shape, with one added wrinkle.

That wrinkle is that input and output are two different cost categories:

• Output tokens already received should never be regenerated;
• But input context may need to be resubmitted, because providers typically don't give you a receipt for cached prompt tokens.

Confusing these two cost types is exactly why bills balloon while the answer stays stuck in place.

Technical Background: Stream Preemption

In LLM inference services, preemption refers to the scheduler forcibly interrupting an in-progress inference task due to resource pressure — such as insufficient GPU memory or queue priority adjustments. Unlike traditional HTTP requests, streaming inference uses Server-Sent Events (SSE) or similar mechanisms to push output to the client token by token. This means that at the moment of preemption, the model may have already generated and transmitted hundreds or even thousands of tokens, and that compute consumption has already been metered on the provider's side — regardless of whether the client received the complete response.

Most major LLM APIs (such as OpenAI, Anthropic, etc.) bill based on actually generated output tokens, not on tokens successfully received by the client. This is the fundamental reason why "resending the entire prompt means paying twice": the output tokens from the first request already incurred a charge, and a second resend generates everything from scratch again, stacking two charges — even though all you needed was the content after what was already generated in the first attempt.

Write a Job Record Before You Call

This approach starts before the call is made, not in the panic of an interruption. Before issuing a request, write a job record with four things:

• An id
• A hash of the prompt
• A token budget
• A zeroed-out cursor

The hash serves as a critical rejection mechanism. It lets you refuse to resume a prompt that has since been edited — if the hash doesn't match, the cursor belongs to a different question, and stitching it in will produce an answer to the wrong thing.

As tokens arrive, don't just flush once at the end — flush at short intervals. The original author captures the difference in one line: flushing only at the end is a successful diary; flushing every few dozen tokens is the failed diary you'll actually encounter. Each flush stores the assistant's generated text so far, the token count you consider spent, and a monotonically increasing revision number. If the process crashes between two flushes, you only lose that gap — and the prefix before the gap is safe.

On Recovery, Construct a Continuation — Don't Start a New Completion

The right move on recovery is to construct a continuation and set the output ceiling to "budget minus spent."

If the API supports assistant prefill, put the stored prefix in and ask it to continue without repeating. If the API doesn't accept prefill, send a brief instruction plus the unfinished tail — not the entire original conversation plus a request to "write a new article."

The original author provides a gate logic implemented in Python. The core is the judgment inside the continuation function:

def continuation(job: dict, floor: int = 32) -> dict | None:
    remain = job["budget_tokens"] - job["spent_tokens"]
    if job["state"] != "open" or remain < floor or not job["prefix"]:
        return None
    return {
        "assistant_prefill": job["prefix"],
        "max_output_tokens": remain,
        "instruction": "Continue from the prefill. Do not repeat it.",
    }

When the remaining budget drops below your configured floor, continuation returns None and you stop — you do not "just in case" resend the original request. None means the prefix has eaten up the useful budget, or there simply isn't a prefix worth continuing. A full replay would ignore the ending you just calculated.

There's also an engineering detail worth noting: the flush writes to a file using a temp-file rename pattern, to avoid leaving a half-written JSON if the process crashes mid-write. But the original author is candid that this is not a distributed lock — two workers sharing the same job id will still overwrite each other's cursors, so each in-flight call needs its own id.

Assistant Prefill Explained

Assistant Prefill is a feature offered by some LLM APIs that allows the caller to pre-supply content for the assistant role in a request. The model treats this as already-generated text and continues from that point, without regenerating it. Anthropic's Claude API has explicit support for this (setting the last message in the messages array to the assistant role without a stop_reason). OpenAI's interfaces achieve a similar effect via the suffix parameter or specific prompt constructions.

The key constraint with prefill is that the model does not guarantee the continuation will be identical to what would have been produced if the stream had never been interrupted — this is fundamentally a new sample drawn from the existing context, with a different random seed and KV cache state. That's why the "when not to use this" section specifically calls out that if your task requires byte-level reproduction of the original sample, this resumption approach does not apply. For most text generation tasks, semantic coherence is usually sufficient and exact reproducibility is not required.

Three Exits: Resume, Abandon, or Spill

The original author emphasizes that you have three exits, and you should choose before the call — not in a panic:

• Abandon: when the remaining amount is below the floor, or the prefix is empty.
• Resume: when the prefix is real and the remaining amount is still within budget.
• Spill: only when the task cannot be discarded and the free slot has already failed — spill to a channel you're willing to wait for.

On spilling, the author is deliberately restrained: spilling is not a loop. One spill, then hand it to a human, or to a queue that can actually reserve capacity. Reservation problems are far older than this cursor, and this article doesn't attempt to solve them.

(Disclosure: The original article was written as part of a MonkeyCode product promotion. Its applicable scope is narrow — MonkeyCode's free model access and free server options can host a "stream interruption drill" after you've read the product's live limits, and only for tasks that can be abandoned. The original article records no quota, credit card, region, or duration numbers, as those figures change and no primary sources were attached.)

The Quieter Trap: Input Replay

The original author points out a quieter, more costly mistake: input replay.

If the provider re-bills the prompt on every resumption, a massive system prompt will eat up all the savings from "not regenerating the prefix." The fix is to separately track input tokens and output tokens on resumption calls. If the input after resumption is roughly equal to the original prompt plus the stored prefix, you've only shifted the cost — you haven't eliminated it.

The solution is either to shrink the recovery context to just the tail you still need, or to simply disable resumption for that prompt shape. A cursor only pays for itself when it prevents a second full completion — it cannot make a bloated prompt cheaper, and it cannot refund tokens the provider has already metered.

Prompt Caching and Input Replay

To mitigate the cost of repeatedly billing large system prompts, some providers have introduced Prompt Caching (such as Anthropic's Cache Control and OpenAI's Prompt Caching). The mechanism hashes prefixes above a certain length threshold; if the same prefix reappears within a short window, the cached KV cache is reused and the cache-hit tokens receive a discount (typically 10%–50% of the original price).

However, this mechanism has several uncertainties: cache TTLs are typically only a few minutes to a few hours; cross-request cache hit rates depend on the provider's load balancing strategy; and both discount rates and cache windows can change at any time without constituting a billing guarantee. This is why the article recommends separately tracking input and output tokens on resumption calls — actively verifying whether input costs were actually reduced, rather than assuming the cache will always hit. If input costs aren't meaningfully amortized, the resumption approach offers diminishing returns for very long system prompts.

When Not to Use This Approach

The author lists several scenarios where you should skip this habit entirely — commendably honest:

• Skip it when you can't persist the prefix outside the process. A cursor that lives only in memory dies with the preemption, leaving behind exactly the replay you were trying to avoid.
• Skip it when the client hides token counts and you can only estimate piece_tokens from string length. A guessed cursor is a lie in your ledger.
• Skip it when byte-level consistency with the original sample is required. A continuation is a new draw, not a resumed sampler — unless the provider documents otherwise. Don't claim a determinism you didn't purchase.
• Skip it for any call that changes world state before the stream ends. Payment captures, email sends, infrastructure changes, customer-facing webhooks — these require idempotency keys, and side effects should run after the text is complete, not inside the stream. If a partial answer is worse than silence, don't pipe it into a channel that might vanish mid-sentence.

Idempotency Keys: The Engineering Concept

An idempotency key is an engineering pattern for preventing side effects from being triggered more than once: the client attaches a unique identifier in the request header, and the server uses it as a key to record the operation's result. If a request with the same idempotency key arrives again — whether due to a network retry or a client resend — the server returns the original result without re-executing the operation. Payment systems (such as Stripe), message queue producers, and email sending services typically require callers to pass an idempotency key.

Mixing LLM stream resumption with downstream operations that have side effects in the same pipeline is a high-risk architectural choice: the stream may be interrupted before triggering the side effect, or after an interrupted resumption it may be incorrectly retriggered, resulting in a payment being charged twice or an email being sent twice. The correct pattern is to strictly decouple LLM text generation from side effect execution — complete text generation and persist the result first, then drive downstream operations with an idempotency key. Connect the two steps via a queue or transaction log, not by chaining them serially inside a stream that can be severed at any moment.

Summary

This habit is small enough to stick with: open a job, flush continuously as text arrives, and on failure ask only for what remains — or ask for nothing at all. Free slots can host this drill, but they can't refund a prefix you've already spent, and they shouldn't become a commitment for work you can't afford to lose. For critical side effects where a partial answer is worse than silence, free capacity was the wrong bet from the start.