Stream Ending Isn't Completion: The Truncation Trap in Free-Model Long Code Generation

When both the model and the server are free, long code generation looks costless—but a hidden failure mode is waiting: the transport layer tells you the stream has ended, yet what you get is a syntactically valid half-function. Over 48 hours of field debugging, one developer rebuilt the collection loop around "verifiable artifacts" and recorded every pitfall and fix along the way.

Why free access changes the shape of the bug

When inference is billed per token, you naturally compress output length and tighten retry budgets. Free models plus free servers invert this incentive: long generations look cheap, so you ask for the entire module and run the client outside your line of sight. The original author states plainly: "Availability was a given, but latency, quota, and durability were things I refused to assume."

The real question isn't "is the model good enough" but "what does a truncated answer look like from the outside." Read timeouts being dropped, proxies closing idle connections, laptops going to sleep—all produce the same thing: a prefix. And a prefix of valid code is often still valid code, which is exactly why it passes smoke tests.

Idempotency and re-runnability are prerequisites for understanding the design in this article. Re-runnability means that when the same task is executed again with identical inputs, the result can be independently reviewed or regenerated without relying on residual state from the previous run. The author emphasizes that "any task that can't be re-run doesn't fit this workflow" because free endpoints can rate-limit or disconnect at any moment. Once a run fails and can't be re-run, the half-captured artifact becomes the only copy, and the hashes and paths in the ledger lose their accountability value. Under the premise of re-runnability, giving each run its own directory and writing a ledger entry for every attempt ensures failures become auditable events rather than irreversible losses.

Three files and one rule

The author's core rule is simple: never append to a file you can't describe. The collector maintains three artifacts in the same directory, each answering one question:

• answer.part: the raw bytes received so far, fsync'd after each chunk.
• ledger.jsonl: an append-only event stream recording chunk offsets, prefix hashes, stream errors, and verification verdicts.
• prompt.txt: the exact request text, written before the first byte arrives.

Why prefix hashes instead of byte counts? A byte count tells you "how much"; a hash tells you "is what you're about to continue writing actually the thing you collected." If a resume writes into a different file than the one hashed, you won't find out until the diff is unreadable.

fsync and write durability: fsync is a POSIX system call that forces data in filesystem buffers to be flushed to physical storage, ensuring that written bytes aren't lost even if the process crashes or the machine loses power. A plain write only guarantees data reaches the kernel page cache, so the last few chunks may be lost on a crash. This article advocates fsync after every chunk—the cost is significantly reduced throughput, but the payoff is an answer.part whose state is honest after a crash. For long code generations on free machines that may be reclaimed at any time, this trade-off keeps "how many bytes have been received" consistent with the actual on-disk content, which is what makes prefix hashes meaningful.

A collector that labels itself honestly

The original article provides a Python collection script, deliberately positioned as a "lab artifact": not validated against your endpoint, so read it and fill in the environment variables yourself before running. The script pulls completions in streaming mode via an OpenAI-compatible interface, with key design decisions concentrated in two places.

First, the [DONE] frame is recorded only as transport_done, never as complete—these are two different assertions. Second, a single malformed frame is skipped rather than raising an exception, because one bad line of data shouldn't invalidate the entire generation.

A validator that rejects "plausible-looking prefixes"

The prompt ends by requiring the model to emit the exact marker # ---END--- on its own line after finishing the code. This sentinel is the heart of the technique: it separates "the model finished speaking" from "the socket closed."

The validator's check order matters critically: check the sentinel first, then hand off to compile(). Reversing the order has a direct consequence—a truncated but syntactically valid prefix sails through compilation and gets delivered with a green checkmark. The author hit a variant at hour 20 of the 48-hour log: a file with balanced braces that terminated inside a string literal fooled the validator.

The syntax-check boundary of compile(): Python's compile() only verifies whether source code conforms to grammar rules; it doesn't verify semantic completeness and doesn't care whether a function body is finished. So a file truncated inside a string literal can pass compilation as long as quotes and brackets happen to balance before the truncation point. The sentinel # ---END--- is a content signal the model actively outputs when generation is complete, orthogonal to syntactic validity. Checking the sentinel first and calling compile() second essentially means using the content-completion signal to rule out truncation first, then using syntax checking to rule out formatting errors—the two cannot be swapped.

Resuming without interrogating the model to death

On failure, the author sends back roughly the last 1200 characters as an assistant turn with a brief instruction: continue exactly from here, no preamble, no summary. At most two resumes; after that the run fails with a non-zero status. Verification verdicts for each attempt are later read from the ledger with jq.

Assistant turns and continuation context: In OpenAI-compatible chat interfaces, messages are divided by role into system, user, assistant, and other turns. Sending the received tail back as an assistant turn during a resume is equivalent to telling the model "you already said this much," so it continues from that position instead of starting over. If you only resend the full prompt without the partial answer, the model generates from scratch, causing the tail to repeat. Setting the resume limit to two and exiting with a non-zero status prevents the model from falling into loops during repeated continuations or producing completions inconsistent with the original answer, while making the failure explicitly visible to the caller.

The 48-hour failure log

In order of occurrence, the original records five major failures:

• Hour 2: Trusted [DONE]. Fix: add the sentinel to the prompt and check it first.
• Hour 9: The resume duplicated the tail because the full prompt was resent without the partial answer. Fix: continuation messages always carry the tail as an assistant turn.
• Hour 20: The validator passed a file that terminated inside a string literal. Fix: sentinel first, parser second—the order cannot be reversed.
• Hour 31: Two answer.part files existed, and the wrong one was hashed. Fix: the ledger records the absolute path alongside every hash.
• Hour 44: A long idle period killed the connection, and the retry loop silently ran four times. Fix: cap resumes at two and exit non-zero when triggered.

The practices the author says they'd repeat without hesitation: a dedicated directory per run, fsync after every chunk, and a verdict line in the ledger for every attempt. The ledger is the sole reason hour 31 took ten minutes to fix instead of a rewrite.

Applicable boundaries and when not to use it

The original uses a table to define the scope. Suitable scenarios include: long code generation that gets human review before use (sentinel, parser, two-resume cap), and bulk docstrings or test scaffolding (same ledger, smaller output). Unsuitable scenarios are equally explicit: latency-sensitive request paths (no SLA to rely on), regulated or confidential data (data-flow review is on you, not the network), and any task that can't be re-run—re-runnability is the entire safety net.

The author also names three types of people who shouldn't use this workflow: those who deploy output without reading it, those who can't afford a second run, and those whose teams won't look at ledger.jsonl when a task fails. If the ledger is just noise to you, this workflow is merely a few extra characters around a truncated file.

What the author deliberately doesn't measure

The original refuses to publish timings, throughput, or success rates, on the grounds that those numbers belong to your endpoint and your prompt shape, not to the author. The script prints its own offsets and verdicts, so your first run will replace every placeholder that could otherwise have been fabricated. The only thing the author can honestly state: across more than one run, the sentinel check caught truncations that compile() alone let through.

The suggested next step is small: if you already have free model access and a free server slot, point this collector at a long generation you already trust, and watch at which hour the verify line flips to false after transport_done.

FAQ

Is [DONE] a completion signal? It's a transport signal. The model's own end-of-output marker is the only evidence that represents content completion.

Does every chunk need fsync? It sacrifices throughput for an honest file after a crash. For long code generation on free machines, the author accepts this trade-off.

Can I skip the ledger if I only run once? Yes, but when the result looks wrong, you'll have no way to tell whether it's a truncated answer or an answer that was simply short.

(Disclosure: the original notes that this article is part of a MonkeyCode product promotion.)