CodeSmith
A system that implements agent constraint engineering, encoding business rules as non-bypassable gatekeepers through a codified rule engine, such as a three-valued decision enumeration and a command arity table, rather than writing them only in prompts.
Core Facts
Timeline (last 90 days)
Verdicts are flushed into the conversation as synthesized <codesmith:runtime_event> user messages, appended only at the tail, and never rewrite the history.
The claim-detection uses contiguous phrase matching rather than regular expressions, because negation words break the phrase, for example "测试未通过" does not contain the substring "测试通过".
The build-category whitelist deliberately excludes bare make and cargo clippy.
Compound commands are split on && and ; and judged segment by segment, and VAR=value environment assignments are stripped off first.
The verdict is written back to the conversation as a four-tuple, which contains the conclusion, the dimension, the evidence location, and the failure type.
Only commands starting with a known test/build prefix are eligible for replay; the test-category whitelist has 20 prefixes and the build-category has 14 prefixes.
Replay follows an approved-replay only safety contract: it uses the conversation's own tool dispatcher, reuses character-by-character the input the model already sent in this round, and grants no new permissions.
The engine-side wiring has only two hook points: the round-end hook scans the final assistant message, and the pre-request flush hook renders buffered verdicts as runtime_event and pushes them into the history.
CodeSmith v0.5.0 implements a test claim verification mechanism; when the model claims a test passed, the engine replays the verification commands the model executed in that round and checks them against the exit code.
This verification mechanism is enabled by default and is implemented in a 914-line module.
26 more timeline events
All Facts (20)
Skill Creator is the core subsystem of Claude Code, essentially an automated framework of "prompt + code generation + file management" that allows users to encapsulate high-frequency operations into reusable structured skill units
75%UnverifiedThe denominator of the claim match rate counts only verifiable claims that are genuinely backed by commands, and Unsubstantiated is never counted in the denominator.
50%UnverifiedThis verification mechanism is enabled by default and is implemented in a 914-line module.
50%UnverifiedThe verdict is written back to the conversation as a four-tuple, which contains the conclusion, the dimension, the evidence location, and the failure type.
50%UnverifiedThe system defines four verdict types: VerifiedPass, VerifiedFail, VerifyError, and Unsubstantiated.
50%UnverifiedUnsubstantiated indicates that the model claims success but never executed any verification-type command in this round, making it the strongest hallucination signal with zero detection cost.
50%UnverifiedThe engine-side wiring has only two hook points: the round-end hook scans the final assistant message, and the pre-request flush hook renders buffered verdicts as runtime_event and pushes them into the history.
50%UnverifiedReplay follows an approved-replay only safety contract: it uses the conversation's own tool dispatcher, reuses character-by-character the input the model already sent in this round, and grants no new permissions.
50%UnverifiedOnly commands starting with a known test/build prefix are eligible for replay; the test-category whitelist has 20 prefixes and the build-category has 14 prefixes.
50%UnverifiedCompound commands are split on && and ; and judged segment by segment, and VAR=value environment assignments are stripped off first.
50%UnverifiedThe build-category whitelist deliberately excludes bare make and cargo clippy.
50%UnverifiedThe claim-detection uses contiguous phrase matching rather than regular expressions, because negation words break the phrase, for example "测试未通过" does not contain the substring "测试通过".
50%UnverifiedVerdicts are flushed into the conversation as synthesized <codesmith:runtime_event> user messages, appended only at the tail, and never rewrite the history.
50%UnverifiedEach verdict is appended to ~/.codesmith/evolution/verdicts.jsonl, recording the timestamp, verdict, original claim text, command, exit code, and model identifier.
50%UnverifiedCreating a high-quality Agent is far more complex than generating ordinary application code, and common defects include: sloppy context management (flattening trajectories into the context, causing loss of KV Cache optimization), lack of discipline in tool design, outdated technology choices, and disconnection from the active ecosystem.
50%UnverifiedCodeSmith's doctor main body is purely deterministic checks and should not be exaggeratedly described as AI self-healing.
50%UnverifiedCodeSmith has a built-in `codesmith doctor` command (the function run_doctor is located at crates/tui/src/main.rs:2310), which can check in one pass the version, configuration files, API key sources, network connectivity, MCP server health, external dependency resolution, and platform sandbox availability.
50%UnverifiedIn crates/agent-runtime/src/dependencies.rs, CodeSmith adopts a 'probe first, then decide' approach to probe Python candidate commands in order (PYTHON_CANDIDATES = ["python3", "python", "py -3"]), and caches the results in a OnceLock for the process lifetime to avoid repeatedly executing the probe on every round of model interaction.
50%UnverifiedCodeSmith v0.5.0's doctor completes the LLM analysis layer (crates/tui/src/doctor_llm.rs): after deterministic checks, warnings/errors are collected and serialized into structured findings, which are handed to a single consultative LLM call to infer root causes and suggest follow-up actions; deterministic results take priority, the LLM only analyzes and does not execute, and on failure or timeout (30 seconds) it is silently skipped.
50%UnverifiedCodeSmith maintains the prompt_zones.rs module to divide stable zones from changing zones, the purpose of which is an engineering constraint made to improve the KV Cache hit rate.
50%