CodeSmith Builds Test-Pass Truth Verification Mechanism in 10,000 Lines of Rust

When a Model Says "All Tests Green," What Is It Really Saying

Any developer who has put an agent to work has lived this scene: the model edits code, runs a test once, then writes in its summary that "all tests pass." You believe it, commit the code, and CI lights up red. Digging through logs reveals the truth—tests ran in a stale directory, failed cases were skipped, or nothing was actually run at all, and the sentence was added out of habit.

This isn't a moral defect; it's the factory setting of a probabilistic mind. CodeSmith v0.5.0 offers an engineering answer: at the moment a conversation turn ends and the model has just declared "all tests green," the engine replays the verification commands the model itself executed during that turn, character for character, and reconciles using exit codes. Verdicts are written back to the conversation as 4-tuples, enabled by default, implemented in a 914-line module.

Commits ab45cdee and 39e77390 together close the gap of "evolution has records but no closed loop." The ROADMAP keeps the ledger: field review found that continuous evolution recorded problems without a closed-loop mechanism, improvement item numbered P3-8. The mistake ledger recorded the debt, MEMORY.md stored the lesson, but the model's "claim" and the world's "fact" were never placed on the same table for reconciliation.

The Design Philosophy of the Verdict 4-Tuple

Four Verdict Types

The output shape looks like this—each verdict carries four elements: conclusion, dimension, evidence location, failure type. In code, it resolves to four verdict types:

• VerifiedPass: Replay passed; the claim holds
• VerifiedFail: Replay failed; the claim contradicts reality
• VerifyError: The replay itself did not complete (timeout / sandbox refusal / tool unavailable); must never be disguised as pass or fail
• Unsubstantiated: The model claims success, but no verification command was ever executed this turn

The fourth is the most valuable signal the entire checker produces. Its detection cost is zero—no replay required, only the observation that the claim has no execution evidence behind it. The comment in the config file is blunter: this is the strongest hallucination signal, because not even a rerun is needed.

Configuration Example

The engine-side wiring has only two hook points. The turn-end trigger condition is a single line: only when result_claim_verifier is enabled and the turn status is Completed does it scan the final assistant message. The flush happens before the request: buffered verdicts are rendered as a runtime_event and pushed into history.

Implementation context for the hook points: The two hook points mentioned are fixed insertion positions in the CodeSmith engine's conversation-turn lifecycle. The so-called "turn-end" hook refers to the callback moment after the model completes a full response and generates its final assistant message—this is the only window for scanning claims, because only after the assistant message is finalized can it be determined what was actually "said." The "pre-request flush" hook sits before the next API request to the model, responsible for serializing and injecting buffered verdicts into history. This "separation of computation and delivery" is not accidental: it ensures the verification logic's latency doesn't block the current turn's completion, and guarantees that verdict injection lands exactly before the next context is consumed, avoiding invalid premature writes. The two hooks have clear responsibility boundaries—one produces verdicts, one delivers them, decoupled in between by a buffer.

The Conservative Boundary of Replay Authority

The most important thing about this module isn't "what it verifies," but "by what authority it verifies." Replaying commands the model ran means the engine executes a shell behind the user's back—why doesn't this cause problems?

The security contract is written in the module docs, and the first item is approved-replay only: replay goes through the conversation's own tool dispatcher, reusing the model's already-sent input character for character—same command, same working directory, same timeout. Verification grants no authority the turn didn't already have: this input already passed through an approval gate once; now it's just being played again.

The second is the eligibility gate. Not just any exec_shell history can be replayed; only commands beginning with known test/build prefixes qualify. The test whitelist has 20 prefixes, the build list has 14, ranging from cargo test to dotnet test. Compound commands are split by && and ; and judged segment by segment; VAR=value environment assignments are stripped first.

The whitelist trade-offs reflect real craftsmanship: the build list deliberately excludes bare make (which would wrongly match make install, make clean) and cargo clippy (that's lint, not build). Background invocations are never replayed.

Claim detection itself has nuance: it uses contiguous phrase matching rather than regex, because "tests did not pass" does not contain the substring "tests pass"—the negation naturally breaks the phrase. One known residual false positive is "not all tests pass," at the cost of one extra replay and an honest exit code in the ledger.

How Verdicts Return to the Conversation

Computing verdicts is only half the work; how to deliver them back is where the design shows its mettle.

Routing is like LSP diagnostics: at turn end, verdicts hang on a pending buffer, and before the next API request they flush in as a synthesized <codesmith:runtime_event> user message, appended only at the tail, never rewriting history.

Why so roundabout? Because append-only conversation history is the lifeline of KV cache. Inserting a verdict in the middle of history invalidates the entire prefix cache; hanging it at the tail forces the model, on its next utterance, to first explain why last turn said all green while the replay came back red.

Honesty of Statistical Accounting

A single verdict is for the model; the long-term ledger is for humans. Each verdict appends to ~/.codesmith/evolution/verdicts.jsonl, one JSON per line: timestamp, verdict, original claim text, command, exit code, and which model said it.

Logging is best-effort: path resolution failure or write failure just logs a WARN and swallows it, because logs must never break the engine that produces events. Malformed lines are skipped on read.

Reading is via /verify stats. The core metric is claim match rate, whose accounting basis is written in the comments: the denominator counts only verifiable claims genuinely backed by commands; the numerator counts those that still hold after replay.

pub fn claim_match_rate(&self) -> Option<f64> {
    let checked = self.verified_pass + self.verified_fail;
    if checked == 0 {
        return None;
    }
    Some(self.verified_pass as f64 / checked as f64)
}

This accounting basis is honest. Unsubstantiated never enters the denominator—mixing "never verified" with "verified and crashed" into one ratio would strip the reading of its ability to distinguish hallucination types. Where a metric draws its denominator exposes what the system wants you to see.

Mechanistic background on KV cache: This touches a key engineering optimization in large-model inference. In autoregressive generation, the model computes key (Key) and value (Value) vectors for every attention layer as it processes the prompt; once computed, these vectors can be cached and reused, avoiding recomputation when the same prefix is encountered and dramatically reducing latency and compute cost. This caching requires strict prefix consistency—once content is inserted or modified in the middle of conversation history, all cache after the modified position is invalidated and must be entirely recomputed. Precisely because of this, any "in-place history edit" incurs severe performance penalties. Appending verdicts at the tail sidesteps this entirely: the original prefix is untouched, cache hits proceed as normal, and new content is only appended at the end. This also explains why the entire system treats "append-only" as an iron law—it is both a correctness constraint and a performance constraint.

The First Domino of the Closed Loop

Threading the mechanism into one line:

1. The turn ends normally; the engine scans the final assistant message, phrase matching catches "tests pass / build succeeded"-type claims
2. It reverse-searches this turn's history for the most recent verification command (20+14 prefixes, segment splitting, env assignment stripping, background invocation exclusion)
3. The command replays character for character through the conversation's own dispatcher—same command, same cwd, same timeout, no new authority granted
4. The exit code settles into one of four verdicts; the 4-tuple renders as a runtime_event hung at the conversation tail, and the model must reconcile next turn
5. The same verdict drops into the jsonl ledger; /verify stats provides categorized counts and claim match rate

Three properties and their respective costs:

On by default—trust is no longer free; the cost is one extra replay per turn for every victory-declaring model, and slow test repos need to weigh the [verification] result_claims switch.

Replay only, zero authorization—the security boundary stays conservative; the cost is that when the model never ran a verification command, the engine can't run one for it and can only report Unsubstantiated.

Verdicts are evidence, not instructions—the engine doesn't rewrite prompts or touch approval gates; the cost is that course correction depends on the model settling its own account, and with a model unwilling to blink, the bill takes a few extra rounds to collect.

The four meta-ability chapters converge here. Looking back reveals they've always stood on the same foundation: verdicts can only hang at the conversation tail, snapshots are append-only, ledgers are append-only—no byte written to context or history may be silently modified.

Why phrase matching beats regex: Claim detection uses contiguous phrase matching rather than regex, and there's an easily overlooked linguistic reason behind it. In natural language, negation is usually achieved by adding a negative word before a verb phrase (e.g., "did not pass," "does not hold"), but there is no stable, regex-precise boundary between that negative word and the positive expression. The example above is typical: the string "tests did not pass" does contain "tests" and "pass," but because "did not" is inserted in between, they don't form the contiguous substring "tests pass," and phrase matching naturally doesn't trigger. Regex, by contrast, would need complex lookahead/lookbehind assertions to handle both affirmative and negative, and easily misses edge expressions. Phrase matching, with its plainest contiguity check, gains free immunity to negation structures. The price is occasional residual false positives, such as the English expression "not all tests pass," which still gets hit—but the system chooses to bear this cost, because one extra replay is far better than missing a hallucination.

One-line Summary

CodeSmith v0.5.0 uses replay reconciliation to make models' test-pass claims answer to exit codes.