Stop Letting AI Replies Impersonate Run Results: 48-Hour Experiment Notes

A Test That Passed, But I Never Ran It

I opened a fresh notebook because a beautifully worded reply always arrives on my side before any process has even started. It looked complete, named a command, and even described a check that had "passed" — a check I never executed. Why was I about to copy that sentence into my log as if it were an observation?

A chat log is not a process table; it cannot tell you which binary actually ran.

That gap is the entire work of this notebook — not a product review, not a leaderboard. I wanted to ask the model for a patch in one place, run a narrow command in another, and read the exit code with my own eyes. If those two records contradict each other, which one deserves the green mark in my notes? I wrote the rules down before asking for the next suggestion, because a fluent sentence can impersonate a result.

First 12 Hours: I Split "Suggestion" and "Run" Into Two Witnesses

Two witnesses are not allowed to grade each other

I split the work into two witnesses that are not permitted to grade each other. The first witness is a free model session, used when I want a draft command or a short explanation without opening a paid console. The second witness is a free server session, used when I need a shell that is neither my laptop nor the model's imagination.

I did not record model names, quotas, machine specs, or promises that "the free tier will still look like this next month." Those details change, and inventing them only lets the notebook lie without blinking. What I can repeat is the order of operations — boring on purpose, and therefore hard to fake: make the request, capture the text, hash the text, run the allowlisted command, and write the exit code next to the hash.

Exit code is an integer status value the operating system returns to the parent process when a process ends; by convention, 0 means success and non-zero means some kind of failure or anomaly. It works as a "witness" because the kernel writes it after the process actually terminates, not because a language model generates it from context — a model can fabricate the words "passed," but it cannot forge the exit code of a process it never started. In the Python mentioned in this article, subprocess.run puts the child process's exit code into the returncode field; sys.exit(n) or raise SystemExit(n) is used to set a return value deliberately. Note that an exit code only reflects "whether this process judged success by its own standard"; it does not guarantee correct behavior: a syntax checker returning 0 only means it found no syntax errors. That is exactly why this article insists on writing "job key + exit code + host identity" on the same line, so a reader can tell which command that 0 belongs to and on which machine it was produced.

One question I kept asking myself

Whenever a reply sounded "already done," I asked: outside the chat, did any process actually start and end with an exit code I can cite? If I could not answer that, the suggestion stayed in the drafts folder, no matter how clean the wording was. A free server is not magic here either, because a free shell may point at the wrong code tree, the wrong user, or a job that never left the queue.

A Logging Script I Would Actually Re-run

The artifact below is a small recorder, not a benchmark and not a declaration that "I timed some vendor." The example paths are just examples; I run them on the hosts I actually have, rather than trusting a pasted success story.

Core design idea: save the model's raw reply to disk first and hash it with SHA-256; then run a command from an allowlist; only when the command's exit code is 0 is it allowed to write "passed." Some people ask: the model says this script has already been run, can I just merge the patch? No — the whole point of this script is to make that kind of laziness slightly harder.

#!/usr/bin/env python3
"""Suggestion ledger. Example workflow, not a recorded benchmark."""

import hashlib
import json
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

ALLOW = {
    "py_compile": [sys.executable, "-m", "py_compile"],
    "unit_check": [sys.executable, "-m", "unittest", "discover", "-s", "tests", "-q"],
}

def stamp(label, text):
    digest = hashlib.sha256(text.encode("utf-8")).hexdigest()
    return {
        "label": label,
        "sha256": digest,
        "chars": len(text),
        "at": datetime.now(timezone.utc).isoformat(),
    }

def run_allowed(key, extra):
    if key not in ALLOW:
        raise SystemExit(f"refusing unknown job: {key}")
    cmd = ALLOW[key] + list(extra)
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
    return {
        "cmd": cmd,
        "code": proc.returncode,
        "stdout_sha256": hashlib.sha256(proc.stdout.encode()).hexdigest(),
        "stderr_sha256": hashlib.sha256(proc.stderr.encode()).hexdigest(),
    }

def main():
    suggestion = Path(sys.argv[1]).read_text(encoding="utf-8")
    job = sys.argv[2]
    record = {
        "suggestion": stamp("model_reply", suggestion),
        "run": run_allowed(job, sys.argv[3:]),
    }
    record["agree"] = record["run"]["code"] == 0
    print(json.dumps(record, indent=2))
    return 0 if record["agree"] else 1

if __name__ == "__main__":
    raise SystemExit(main())

The command itself should be boring, and boring is a feature

The command I actually want in my notebook is boring, and that is a feature rather than a lack of ambition. Save the raw reply first, then compile, then determine whether the unit test discovery is really on this host's allowlist. The model can propose an install command with a pipe in it, but the recorder will not sprout a new key just because that text sounds urgent. Why should a draft have the right to invent shell history?

mkdir -p notes/day1

**SHA-256 hash** is a cryptographic hash function that maps an input of arbitrary length to a fixed 256-bit (32-byte) hexadecimal string, with avalanche effect and collision resistance: change one character of the input and the output is almost completely different, and within practically feasible bounds it is impossible to find two different inputs producing the same output. This article uses it for two things: first, hashing the raw text returned by the model as proof that "what I received is exactly this text"; second, hashing the variants of a command's stdout and stderr to judge whether two runs produced the same output. A hash can only prove **integrity** — that the text was not silently replaced or truncated — not that the content is correct or wise. Writing the hash next to the exit code simultaneously pins down "which claim" and "which execution," so a later person cannot overwrite the old record with a similarly worded new reply.
# Save the raw completion yourself. Do not let the model write this file.
cp reply.txt notes/day1/reply.txt
python3 ledger.py notes/day1/reply.txt py_compile path/to/changed_module.py
echo "ledger_exit=$?"

When the server and my laptop do not share the same code tree, I keep a second command around. Before trusting a "0," I print the hostname, user, and executable path, because a green exit code on the wrong machine is just another story. This check cares about the identity of the run, not about how I feel regarding the responsiveness of a free machine.

python3 - <<'PY'
import getpass, platform, sys
print(platform.node(), getpass.getuser(), sys.executable)
PY
# Sketch only: replace the alias with the host your panel actually shows.
ssh free-host 'python3 -m py_compile path/to/changed_module.py; echo exit:$?'

That ssh line should be labeled a sketch. The host alias free-host is a placeholder, not a hostname I measured, and your free server panel may use another name. If the panel does not give you a shell, the notes on this branch stop there. The free model can still draft a patch, but it cannot replace the missing exit code.

What Happened When I Let the Reply Grade Itself

The social breach

The first breach is social, not technical, and it appears the moment a paragraph says "the check has passed." The server job has not started yet, but my fingers already want to write "verified" next to the suggestion's hash. Who am I trying to impress — a future reader, or my own impatience? I delete that word and leave only the hash, until ledger_exit actually exists.

The command does not match, but looks relevant

The second breach is a command mismatch that looks close enough to fool a tired note-taker. The reply describes unittest, while the only allowlisted task I am willing to run that hour is py_compile, so a clean compile is not the test that paragraph claims. The compiler's 0 cannot prove the test passed, and the model's summary cannot prove the compiler ran. So I started writing the job key and the claim on the same line, so a later reader can see the gap without guessing.

The silent failure at the environment layer

The third breach is the environment, which is the one I am most willing to warn about repeatedly. A free server may start with a different working directory, a different user, or a queue that never dispatched the command I thought I had sent. If the ledger file is missing, I do not infer success from the silence, because silence is exactly how a dropped job disguises itself. Did the process start, did it end, or did I merely receive a sentence about "ending"?

Allowlist here refers to an explicit, predefined set of executable tasks: only tasks whose key is in the ALLOW dictionary may be started by the recorder, and passing in an unknown key exits immediately with an error. It is the opposite of a blocklist — a blocklist says "anything but these," while an allowlist says "only these," denying by default every command not listed. This design targets two risks: command injection and trust drift. If the recorder directly executed arbitrary commands or arguments from the model's reply text, a seemingly urgent install instruction could actually be run; and if new task keys could be added ad hoc mid-session, the suggester would effectively be able to retroactively authorize its own permissions. The rule in this article is to write allowlist changes into a separate note and route them through a separate process, thereby separating "drafting" from "authorizing execution" in time. An allowlist does not guarantee the listed commands are safe or relevant; it only guarantees that unapproved processes will not start because of a fluent sentence.

A Decision Table I Can Reuse Tomorrow

I do not want to give a score that pretends to measure model quality, because I never built that experiment and I will not fake rows. This table only decides whether my notebook can say "ran" or must say "draft." The columns are evidence I can point to, not a feeling about how confident the wording sounds.

What I have	Reply only	Server log only	Both, and exit code agrees with the claim
Notebook status	draft	incomplete	candidate
Can I say verified	No	No	Only for that job key
Next step	Save hash, do not patch	Re-run or fix host	Read the diff, then decide
Expected failure	Fluent fake pass	Wrong code tree or never dispatched	Diff still needs review

Candidate is not merge. It only means a command in my allowlist exited in a way that matches that narrow claim, and the host is one I can name in the notebook. If the reply promises tests and I only compiled, that line stays incomplete, no matter how happy the compiler is. Would I expand the allowlist mid-session because the model sounded nice? No — that change goes into another note, after I read the new command.

What I Will Repeat, and What I Will Not

I will repeat the capture step, the allowlist, and the refusal to write verified without a ledger_exit in the same folder. I will also repeat the host line, because a free server is only useful if I can distinguish it from my laptop and from the chat. Free model access is the part I keep for drafting, especially when I want to rephrase a stack trace before opening a paid seat. The free server option is the part I keep for running, because a suggestion that never meets a shell is still only a suggestion.

I will not repeat any sentence that cites quotas, GPU names, or latency numbers I did not measure that day. I will not pipe model commands into a shell, and I will not let the recorder accept arbitrary arguments from the reply text. I will not use this notebook to judge which vendor is best, because two availability switches are not a leaderboard. If your work needs private datasets, signed production hosts, or a change window with an on-call owner, do not rehearse that change on a free shared server just because the reply looks ready.

Who should skip this workflow

The people who should skip it are easy to name, and I would rather name them than pretend this ledger suits every team. Skip if you cannot read the server logs yourself. Skip if policy forbids sending even sanitized stack traces to an external model. Skip if the patch touches credentials, production data, or a service you cannot roll back before your next coffee. Skip if you need compliance proof — field notes are not an audit, and a free tier is not a contract I am happy to invent out of thin air.

The Limits I Left in the Notebook

This workflow does not prove the model understood the code, nor does it prove the server matches production. The 120-second timeout in the example is a sketch's guardrail, not a service level I negotiated with anyone. py_compile catches syntax, not behavior; unittest discover only helps if those tests already exist and already mean something. A hash tells me whether the text changed, not whether the text is wise.

I also cannot tell you whether the list of free models or the free server button will still be in the same place next week, or whether the limits will be the same. Look at the console you actually have, write down what you see, and do not copy these notes as if they were a status page. If the free server never gives you a shell, stop at draft and run the allowlisted command somewhere you control. A missing witness is a result, and it should look like a missing witness in the file too.

If you already use MonkeyCode, the useful part is ordinary: take the free model path when you need a draft, take the free server path when you need a shell, and keep their outputs in separate files. But before shipping anything, I will still read the diff myself, because agreement on an exit code does not mean the change is the one I had in mind. The point of this notebook is to let my future self see what ran, what never started, and what I refused to call done.