Building a Response Receipt: A Tiny Script to Intercept Blank and Hallucinated AI Output

Late at night in the lab, just a few hours before a demo, a free model server returned three spaces — and those three spaces nearly got stuffed into a portfolio README as a result. That's a true story from this article's author, and it raises a question worth asking for anyone using large language models: Are we blindly trusting model output just because the text box used to work?

This article introduces a minimalist engineering habit: "freezing" a single model response into a verifiable JSON receipt card, then using a Python script of fewer than fifty lines to intercept blank or off-topic answers before they make it into your work.

Why You Need a Receipt, Not a Screenshot

The author's core argument is sharp: screenshots lie. "A screenshot smiles at you in the moment, and betrays you a week later when the same prompt returns nothing."

A receipt, by contrast, is more like a bus ticket. It can't prove the city was beautiful, but it proves you boarded with a specific note in hand. In other words, a receipt captures a reproducible intermediate state — not the vibe that something "seemed to work."

This approach transforms class notes or project requirements into machine-checkable artifacts. The author's own project requirements were dry — "the demo must show a text receipt, not just a color badge, and sample files must be CC0" — but that dry sentence becomes the entire basis for judgment. If a model's answer can't be tied back to it, the answer doesn't deserve to be called a result.

What the Script Does (and Doesn't Do)

The goal is clear: a single command reads a JSON card and exits with a non-zero status code when the answer is missing, blank, or has no meaningful word overlap with the context. Failures must appear in the terminal.

The author emphasizes an honest pairing: a green README badge next to a red process exit code. The script is a gate, not a grade — even if the receipt passes, you still have to read the sentence yourself.

Core Logic

The script's dependencies are intentionally minimal: Python 3.11+, a terminal, and zero third-party packages. Even if a classmate clones the folder on a train with no signal, the check can still run.

Validation happens in three layers:

• Field completeness: prompt, context, answer, and source are all required. Any missing field returns missing_fields.
• Blank detection: After calling strip() on the answer, an empty string returns empty_answer. Three spaces are not "a polite pause" — they're an explicit failure.
• Grounding detection: The answer and context are each tokenized and their intersection is checked. If no shared words exist, the result is ungrounded.

The stopword list (STOP) design is worth noting. High-frequency words like the, and, not, and only are excluded as evidence so they can't masquerade as overlap. Words shorter than 3 characters are also filtered out.

A note on "grounding": The concept comes from grounded language understanding in NLP. The core idea is that a valid response must have a traceable connection to the given context or knowledge source — not be generated out of thin air. In retrieval-augmented generation (RAG) systems, this connection is typically measured via vector similarity or cited passages. This script uses an extremely lightweight proxy: lexical overlap — directly comparing the tokenized sets of the answer and context. No embedding model or network request is needed, but the tradeoff is that only surface-level word signals are captured; semantically equivalent paraphrases are invisible to it. The author explicitly acknowledges this in the "limitations" section: lexical overlap is a necessary but not sufficient condition for grounding. It can catch obvious hallucinations, but cannot determine whether a correct paraphrase is truly faithful to the source.

Three Cards, Three Verdicts

The author demonstrates the script's behavior with three fixture files — the most convincing part of the article.

Card one: grounded.json. The answer restates claims that actually appear in the notes. Exit code is 0. Overlapping words include demo, show, text, receipt, color, and badge. The author reminds us: the source field is just a label you typed manually — it's not a server signature and shouldn't be read as one.

Card two: blank.json. The answer is three spaces, with source set to live-free-server. This is the card that nearly made it into the README. The script returns empty_answer with exit code 1. The author is unequivocal: "Silence is not a summary. I didn't retry three times and average out the feeling."

Card three: a fluent lie (ungrounded.json). The answer says "this model scored 99 on a private leaderboard" — grammatically correct, but sharing no content words with the project notes. Exit code is also 1.

"If your first draft only checked that the string was non-empty, this one would pass easily. That was my first draft. It felt kind. But it was wrong."

This line names the critical trap: a non-empty string is precisely how demos most often fool us. The leaderboard sentence sounds like a result only because it contains a number. As the author puts it — "numbers are costumes." 99 never appeared in the project notes. The script doesn't care whether the sentence is grammatical.

Where This Method Breaks Down

The author doesn't dodge the method's limitations — which is exactly what makes the article credible.

The echo problem: If you copy the context directly into the answer, the receipt will applaud — but that's an echo, not understanding.

The paraphrase problem: A clean, correct paraphrase that shares no content words will get flagged as a failure. The author walked into this trap personally: paraphrasing "visitors need text, not green dots, and sample data is in the public domain" caused a failure because it didn't share the words receipt, badge, or CC0.

This leads to a precise framing: the lexical gate is a smoke alarm, not a fire inspector. It checks for word overlap with the notes you provided — it does not check truth, licenses, or whether a paraphrase is fair.

Who should skip this method? If you need citation verification, grading, or anything that might be used as an evaluation framework by a hiring manager, this is the wrong tool. If you need the server to stay online during a demo, you need a different solution. If your context is shorter than one sentence, lexical overlap becomes a coin flip.

The author also lists common mistakes they've personally made: treating blank output as an answer; printing JSON while ignoring the exit code; letting the count as evidence before the stopword list existed; storing API keys in fixtures (never do this); and most dangerously — retrying until a pretty sentence appears, then freezing that pretty sentence with no notes attached. "That's how a demo becomes fiction."

On "fluent lies": What the author describes has a formal name in LLM research: hallucination — when a model generates content that is grammatically fluent and confidently phrased, yet inconsistent with facts or the given context. Hallucinations are dangerous precisely because they're hard to distinguish from real output on the surface: both are non-empty strings, both have complete sentence structures, and both may include numbers or citations. Current mainstream mitigation techniques include retrieval-augmented generation (RAG), reinforcement learning from human feedback (RLHF), and post-generation fact-checking pipelines. The lexical gate in this script can be understood as the lightest-weight local alternative among these approaches — it requires no external services, only the context notes you provide. This gives it unique practical value for offline scenarios and rapid demo validation, while also inheriting the limitations common to all lightweight solutions.

An Extension Worth Trying

The author leaves an open exercise: change the passing rule so that ok requires at least two overlapping words, then rerun the grounded card. Do text plus receipt still count as grounded, or are you just rewarding a copy-paste?

Then add a fourth fixture whose answer is a genuine paraphrase with zero shared words, and decide: do you want a failure, or do you add a manual note to the source field?

The script itself isn't complex. But the engineering intuition behind it is worth spreading: treat the output of a large language model like a record with fields, not like a feeling. Free model services are fine for the one live call that generates the card, but they're a poor long-term dependency — when someone opens your README on a train, the service may already be gone. Freezing results into verifiable receipts is how you build a portfolio that holds up over time.