All Safety Tests Passed, Yet Product Tests Failed: A Statewave Guide Experiment Retrospective

AI-generated product knowledge can be technically fully correct and still completely useless to users. The Day 4 experiment log from the Statewave Guide project delivers a counterintuitive conclusion: once the model stops making things up, the real problems are just beginning.

Safety Tests: A Clean Sweep

The experiment setup followed the previous day: a single model anthropic-opus-5, targeting an application written by the team itself, checking only for patterns they already knew to look for. The results were near-perfect — zero instances of incorrect capability claims being accepted, zero instances of unsupported claims being accepted, model restraint at 30/30, and a verified claim output rate of 95%.

These numbers show that, at least when it comes to "keeping the model from saying wrong things," the verification-based architecture has solidified its footing. Any claim that couldn't be traced back to the application's actual capabilities was blocked by the system. There were no gaps in the safety defense line.

The problem is that safety does not equal usefulness.

Model restraint refers to the model actively choosing not to give a definitive answer when facing questions lacking supporting evidence, rather than forcing out a plausible-sounding answer. It measures the model's ability to "hold back": when it cannot trace and confirm from the application's actual capabilities whether a certain feature truly exists or whether a certain permission is available, the model should refuse to assert, rather than filling the gap with fluent language.

In a verification-driven architecture, model restraint is the opposite-pole metric to hallucination rate. A high hallucination rate means the model said things it shouldn't have; a high restraint rate means the model stayed silent when it shouldn't have answered. 30/30 means that across all test scenarios requiring restraint, there was not a single overstep. But what warrants caution is this: a perfect restraint score does not necessarily lead to usefulness, because both "saying nothing" and "saying things people can't understand" can leave users with nothing.

Product Tests: A Failing Grade at One-Third

The team then asked a second question: do these outputs actually help people?

Scoring was not done by humans. An independent model, OpenAI GPT-5.6 Sol, scored each of the 21 feature points one by one according to a fixed rubric. It was not the model under test, so this doesn't count as self-evaluation, but it also isn't human review — the team explicitly acknowledged this limitation in the original text.

The result: median usefulness of 1/3, with only 24% rated as "truly useful."

No dangerous errors, and nothing completely useless. But almost everything was mediocre. The team originally thought the remaining problem was a capability gap, but the data gave a different answer:

1.43   Average score for features with verified capabilities
1.14   Average score for features without verified capabilities

The gap was negligible. In a later round of experiments, after fixing language issues, the same comparison became 2.00 versus 1.07, and the capability signal finally emerged — meaning the signal had been there all along, just obscured by poor presentation.

Using another model (OpenAI GPT-5.6 Sol) as a judge falls under the common LLM-as-a-judge evaluation paradigm. Its core assumption is that there is a degree of independence between different model families, so using model B to judge model A's output can avoid the systematic preference that arises when a model scores itself — namely, self-evaluation bias.

The limitations of this method are also obvious. The judging model may still have inherent preferences for certain writing styles, terminology density, or sentence structures, and these preferences may not align with real users' understanding. It cannot perceive the context of users' actual usage scenarios, nor can it reproduce the subtle human trade-offs between "verbose but clear" and "concise but confusing." Therefore, the 24% figure should be understood more as a relative comparability signal than an absolute usability measure. The team introduced it to obtain a directional data point in the absence of large-scale human annotation.

Smoking Gun: Debug Statements Read Directly to Users

The most glaring finding was this: 14 of the 21 outputs were judged as "too technical," and 13 were flagged as missing a certain capability.

The real problem came down to a single sentence. 10 feature points directly exposed this raw text to users:

"No capability, route or permission has been verified for this feature."

All 10 of these feature points, without exception, were rated as "too technical." 10/10.

This isn't product copy — this is a debugger having an existential crisis in front of the customer.

Another feature point's evidence chain was exemplary — POST endpoint, create permission, submit pathway, button, all complete — yet the determinism renderer proudly output this sentence:

"You can create a new invoices create form and submit the invoices create form form…"

Technically impeccable, linguistically dead.

What this exposes is the boundary problem between the Renderer and internal diagnostic information. In a verification-driven pipeline, the system internally needs to distinguish three states: capability verified, capability unverified, and capability explicitly denied. When a feature point falls into the "unverified" state, the system internally generates a diagnostic record for later tracing and debugging.

The problem is that some rendering paths have no user-facing alternative copy prepared for the "unverified" internal state, so they directly output the diagnostic record's raw text as the copy. From an engineering perspective this is reasonable — the field has a value, the format is legal, the logic is consistent; from a product perspective this is a disaster, because what the user sees is not a readable instruction like "this feature is not yet confirmed; we suggest contacting the administrator," but rather a meta-description pointing to the system's internals. This shows that a layer of conversion logic is missing between the verification layer and the presentation layer, and that layer is precisely what the Guidance Compiler described later is meant to fill.

A New Layer in the Architecture: From "What We Know" to "What Should Be Said"

These two cases point to the same boundary. The team had previously separated "what the application actually does" from "what the model thinks it does." Now they needed to separate one more layer: "what we know" and "what humans actually need to hear."

So the architecture gained a new layer:

Product Model        →  What do we know?
Guidance Compiler    →  What is worth telling the user?   ← new
Renderer             →  How should it be said?

Verification rules were not relaxed, and the model's permissions were not expanded. The team simply finally acknowledged something obvious to software developers: a technically correct data structure does not automatically know how to talk to humans.

The responsibility of this Guidance Compiler layer is to translate verified product facts into guidance that users can understand and use, while introducing no new, unverified claims during the translation process. This is a key architectural turning point in the entire series.

The Guidance Compiler sits between knowledge representation and natural language generation. It receives verified structured product facts (for example, that a certain button exists, a certain permission is confirmed, or a certain pathway can be submitted), and outputs user-facing executable guidance. It must operate under two constraints: first, it must not introduce any new, unverified claims; second, it must translate technical field semantics into descriptions of actions and consequences that users can understand.

This differs from traditional template rendering. Template rendering cares about how to fill fields into fixed sentence patterns; the Guidance Compiler cares about, given the current set of facts, what information is most worth saying to the user, at what granularity, and whether anything should be omitted. It is similar to converting a database schema into natural language explanations, but with two added product constraints: "selective expression" and "avoiding technical exposure." Adding this layer architecturally means the system acknowledges that an explicit conversion step is needed between verification completeness and expression usability, rather than relying on the renderer to handle it incidentally.

Three Judgments Worth Noting

Safety and usefulness are two independent axes. A system can simultaneously achieve "zero incorrect claims" and "almost everything mediocre," and there is no automatic conversion relationship between these two things. Getting verification to a perfect score does not mean the user experience will naturally improve.

The scoring method itself needs to be questioned. Using another model to judge the output of the model under test can only eliminate self-evaluation bias; it cannot replace human evaluation. How much reference value the 24% figure has depends on the design of the scoring rubric, and the team remained clear-eyed about this.

The problem may lie in the presentation layer, not the capability layer. In the first round of experiments, the capability signal was almost invisible; after fixing language issues, the signal became clear. This means early experiments can easily misjudge "poor expression" as "insufficient capability," and extra care is needed when making product judgments.

Statewave Guide is built on top of Statewave — an open-source, self-hostable AI agent memory runtime focused on governance and source traceability rather than mere retrieval, under the Apache 2.0 license. The next question the project needs to answer is: can verified product facts be transformed into guidance useful to people without fabricating any new content?

One-line Summary

AI product knowledge can be error-free yet useless; the architecture needs a new Guidance Compiler layer to translate verified facts.

Paragraph Summary

Statewave Guide's Day 4 experiment reveals a counterintuitive conclusion: once the model stops fabricating, the real problems begin. Safety tests all passed — zero incorrect claims, model restraint at 30/30, verified output rate of 95% — but in product tests the median usefulness was only 1/3, with just 24% rated truly useful and almost everything mediocre. The smoking gun: 10 feature points directly exposed debug statements to users, 100% rated as too technical. The team realized safety and usefulness are two independent axes, and the problem may lie in the presentation layer rather than the capability layer — the capability signal only emerged after language issues were fixed. The architecture therefore added a Guidance Compiler layer to transform verified product facts into human-understandable guidance without introducing new unverified claims.