Replay Cassette
A testing tool for recording and replaying LLM API requests and responses. By caching HTTP interactions, it enables offline testing that is zero-cost and reproducible.
Timeline (last 90 days)
A developer spent 48 hours building a "Replay Cassette" tool for recording and replaying LLM interactions offline, to test prompts and parsers at zero cost.
The Replay Cassette tool is implemented using only the Python standard library, works with any HTTP model endpoint, and does not depend on a specific vendor SDK.
The author defines three contracts for the replay cassette: replay runs must cost zero; misses must fail loudly (exit code 2 and print the fingerprint); cache keys must be explainable.
A cache that silently falls back to a network request on a miss masks regressions, whereas the replay cassette errors out and exits on a miss to expose regressions.
The storage format chosen is append-only JSONL, which withstands hard shutdowns better than repeatedly rewriting a JSON file and is also easier to grep and diff.
At hour 27 the author discovered the system prompt never entered the hash, because the system message was injected by a thin client wrapper reading an environment variable, causing two different prompts to produce the same cache key.
Chunk boundaries should not be persisted in streaming responses, because chunks are merely a transport-layer artifact and providers may change how they split; only the assembled text and finish reason should be stored, and re-chunked by a fixed chunk_size on replay.
The author's practical decision rules: record once and replay many times while shaping prompts or parsers; replay directly when only post-processing code changes; record separately and key by alias when comparing model aliases; make real calls when checking the provider's current behavior; use replay for CI regression gates; use real calls for load or latency testing.
The replay cassette answers "did my code change" but never answers "is the model good today," because it is only a frozen observation and cannot validate quality.
Responses containing user data should not be stored in the cassette unless there is a trustworthy scrubbing step, because append-only logs are extremely good at preserving secrets forever.
2 more timeline events
All Facts (12)
It is suggested that in recording mode you send the same payload five times and report how many different answers come back, as a "stability probe" to quantify the proportion of noise in your improvements.
50%UnverifiedThe Replay Cassette tool is implemented using only the Python standard library, works with any HTTP model endpoint, and does not depend on a specific vendor SDK.
50%UnverifiedThe author defines three contracts for the replay cassette: replay runs must cost zero; misses must fail loudly (exit code 2 and print the fingerprint); cache keys must be explainable.
50%UnverifiedA cache that silently falls back to a network request on a miss masks regressions, whereas the replay cassette errors out and exits on a miss to expose regressions.
50%UnverifiedThe storage format chosen is append-only JSONL, which withstands hard shutdowns better than repeatedly rewriting a JSON file and is also easier to grep and diff.
50%UnverifiedResponses containing user data should not be stored in the cassette unless there is a trustworthy scrubbing step, because append-only logs are extremely good at preserving secrets forever.
50%UnverifiedIf you call an LLM only once a day, or the only question you repeatedly ask is whether the model is currently reachable, this replay cassette tool is not worth using; when you have a dozen use cases each run dozens of times, it pays for itself in about two hours.
50%UnverifiedChunk boundaries should not be persisted in streaming responses, because chunks are merely a transport-layer artifact and providers may change how they split; only the assembled text and finish reason should be stored, and re-chunked by a fixed chunk_size on replay.
50%UnverifiedThe author's practical decision rules: record once and replay many times while shaping prompts or parsers; replay directly when only post-processing code changes; record separately and key by alias when comparing model aliases; make real calls when checking the provider's current behavior; use replay for CI regression gates; use real calls for load or latency testing.
50%UnverifiedThe replay cassette answers "did my code change" but never answers "is the model good today," because it is only a frozen observation and cannot validate quality.
50%UnverifiedAt hour 27 the author discovered the system prompt never entered the hash, because the system message was injected by a thin client wrapper reading an environment variable, causing two different prompts to produce the same cache key.
50%UnverifiedA developer spent 48 hours building a "Replay Cassette" tool for recording and replaying LLM interactions offline, to test prompts and parsers at zero cost.
50%