Unstable LLM Calls? Build a Replay Cassette Recorder in 48 Hours

Why Your Prompt Tuning Keeps Costing You Money

Send the same prompt to a large model, and ten seconds later the wording of the reply has changed. Sampling settings explain part of it, batching and server routing explain the rest—but none of this helps you judge "improvement or noise." This is the dilemma many developers face every day: you want to verify whether a new system prompt actually improves output quality, but with no frozen baseline, every comparison costs you a real API call.

One developer spent 48 hours making a decision: stop tuning prompts and instead build a "Replay Cassette." The core logic is simple: record every interaction with the model, then replay all tests offline—no money spent, no impact from network jitter. According to the technical blog, the entire solution is implemented using only the Python standard library and works with any HTTP model endpoint.

Three Iron Rules: Replay Must Be Free, Misses Must Error

Before writing any code, the author wrote down three contracts, because a caching layer corrupts once its contracts become vague:

1. Replay runs must be zero-cost—when fixing a single regex in the parser, the same test suite should be able to run forty times without a single real call.
2. Misses must fail loudly—when a recorded response is missing, the test framework exits with code 2 and prints the fingerprint; it must never silently fall back to a network request.
3. Cache keys must be explainable—you should be able to print the fingerprint next to the request and see exactly which field changed.

The second rule is the one most people skip, yet it is precisely what turns a "cache" into a "testing tool." A cache that secretly calls the network on a miss will mask the very regressions you are trying to catch.

Core Implementation: A Dependency-Free Python Cassette

The entire tool is written in standard-library Python. It wraps your own call function, not a vendor SDK, so it works with any HTTP model endpoint. The storage format is append-only JSONL, which survives hard shutdowns better than repeatedly rewriting a JSON file, and is also easy to grep and diff.

The key design is generating the cache key: a canonicalized SHA-256 hash of the request payload. But here lies the first pitfall—the author's initial fingerprint hashed the raw request dictionary directly, causing every replay to miss. There were three reasons: different dictionary insertion order, trailing whitespace drift in message content, and implicit defaults such as temperature ({"temperature": 1.0} and {} are the same request to the server but two different keys to the hash). The fix was to sort keys, use compact separators, strip trailing spaces line by line, and merge defaults.

The author concludes: a cache key is an API contract you sign with your past self. Write the canonicalization function first, pin it with a version field, and bump the version whenever the semantics change.

SHA-256 and Canonical Hashing

SHA-256 is a cryptographic hash function that maps an input of any length to a fixed-length 256-bit (64 hexadecimal character) digest, with an avalanche effect—change even a single bit of the input, and the output becomes unrecognizable. This makes it naturally suited for cache keys: as long as the request semantics are the same, the hash should be identical; if the semantics differ, the hash must differ. But the hash function itself does not understand "semantics"; it is only sensitive to byte sequences. Therefore "canonicalization" becomes the critical step: before hashing, logically equivalent requests must be reduced to a unique byte representation—for example, sorting dictionary keys, using compact separators to eliminate whitespace differences, and merging implicit defaults. Otherwise, two requests that are completely identical to the server will generate different fingerprints, causing the cache to always miss. The author's emphasis on "write the canonicalization function first and pin it with a version field" is precisely because in a cache system, once the key generation rules change, all historical record keys become invalid; this evolution must be managed through an explicit version number.

JSONL Append-Only Storage

JSONL (JSON Lines) is a text format with one independent JSON object per line. Compared with a format that is a single large JSON array, it supports appending entries one by one without rewriting the entire file. This is crucial for a "replay cassette": the recording process may be interrupted by a crash or power loss, and append-only writes lose only the last incomplete line, whereas rewrite-based storage may corrupt the entire file. In addition, JSONL is naturally suited for grep, diff, and streaming reads, making manual inspection and version control easier.

The Lesson at Hour 27: The System Prompt Never Entered the Hash

A more insidious bug appeared at hour 27: the author changed the system prompt, but the replay returned the old answer and all tests were green. The fingerprint looked normal; the problem was in the test harness—the system message was injected by a thin client wrapper that read an environment variable, so the hashed dictionary contained only the user turn. Two different prompts produced the same key, and the replay confidently returned the first answer.

The fix used two guards that should have been written on day one: explicitly assert that the first message in the message list has the system role; and write the byte hash of the prompt file into the key. This lesson generalizes to any project: never let a library assemble the request being hashed for you. Build the message list yourself and assert its expected shape.

Thin Client Wrappers and Implicit Request Assembly

Many model SDKs or internal client libraries offer "convenient" wrappers: you pass only the user message, and the library automatically injects fields such as the system prompt, temperature, and max tokens from environment variables, configuration files, or default parameters. This design is convenient for normal calls but extremely dangerous when building cache keys—because the hashed dictionary contains only what you explicitly passed in, while the request actually sent to the server also contains content the library quietly added. Two different system prompts may generate exactly the same user message dictionary, thereby colliding on the same cache key and returning the wrong historical response during replay. The fix is not to patch the wrapper but to bypass it entirely: build the complete message list yourself (including the system role), explicitly assert its shape, and include the byte hash of the prompt file in the key. This lesson applies to any scenario that requires reproducible requests: the object being hashed must be a byte-for-byte representation of the online request that you fully control.

Streaming Responses: Don't Store Chunk Boundaries

Streaming output is another trap. The author's first instinct was to persist chunk boundaries so that the replayed stream looked byte-identical to the original—a mistake, because chunks are merely transport-layer artifacts, and the provider could well split things differently tomorrow.

What the parser really needs is the assembled text plus the finish reason, delivered through the same iterator interface. On replay, just re-split using a fixed chunk_size, so the streaming path is genuinely exercised on every replay and the parser sees real partial chunks. The author's advice is direct: if your parser depends on the exact chunk shape, that is a problem with the parser and should not be hidden in a fixture.

Transport-Layer Artifacts and Parser Contracts

A "chunk" in a streaming response is an implementation detail of the HTTP transport layer or the provider API, not a semantic unit. The same complete reply may be split by the provider by character, by word, by sentence, or by a fixed number of bytes, and the splitting strategy may even change between versions. If a test fixture persists the original chunk boundaries, the parser implicitly depends on those boundaries; once the provider changes its splitting method, the parser crashes on real calls while replay tests still pass—exactly the manifestation of "a cassette is a snapshot, not the truth." The correct approach is to store only the assembled complete text and the finish reason, re-split by a fixed size on replay, and deliver through the same iterator interface as the real stream. This way every replay genuinely exercises the streaming parsing path, and the parser can encounter edge cases such as partial chunks and empty chunks, thereby continuously exercising its robustness.

When to Record, When to Call Directly

The author provides a practical decision table: when shaping prompts or parsers, record once and replay many times; when only changing post-processing code, replay directly; when comparing two model aliases, record separately and key by alias; when checking the provider's current behavior, you must make real calls with multiple samples—a cassette is a snapshot, not the truth; for CI regression gates, use replay and exit with code 2 on a miss; for load or latency tests, always use real calls, because replay tests your disk, not the network.

The core distinction: replay answers "did my code change," never "is the model good today." Confuse these two questions, and a team will cling to stale fixtures for months.

Applicable Boundaries and Who Should Not Use It

A cassette is a frozen observation and cannot validate quality—it will happily pass even if the provider swaps the model behind an alias. Do not store responses containing user data in a cassette unless you have a trustworthy scrubbing step—append-only logs are extremely good at preserving secrets forever. Teams doing continuous evaluation should use a real evaluation pipeline, not a local cassette.

If you make only one call a day, or the only question you repeatedly ask is whether the model is currently reachable, this tool is not worth it. When you have a dozen use cases, each run dozens of times, it pays for itself in about two hours.

The author also suggests adding a "stability probe" early: in recording mode, send the same payload five times and report how many different answers came back. That single number tells you how much of your so-called improvement is noise.

FAQ

Q: What is the difference between a Cassette and an ordinary response cache? A: An ordinary cache falls back to the network on a miss; a Cassette errors out and exits on a miss. The former masks regressions; the latter exposes them.

Q: Can it be used in production CI? A: Yes, but it is recommended to commit only a small, scrubbed seed cassette for the smoke suite, and keep the full cassette outside the repository.

Q: Why not just mock the API call? A: Mocking requires you to hand-write fake responses, whereas a Cassette records the real output of the real model, including formats you never expected to appear.

One-line Summary

A developer spent 48 hours building a zero-dependency replay cassette that makes prompt-tuning tests offline, free, and fail-on-miss.