Truth Isn't Relevance: Statewave Guide Exposes Blind Spots in AI Agent Verification

In the development of AI agents and code comprehension tools, ensuring the factual accuracy of model outputs has long been regarded as a core safety metric. However, recent experiments from the Statewave Guide project show that factual correctness alone may be far from sufficient—a factual statement about form submission passed every verification rule, yet had nothing to do with the feature the user was actually asking about. This kind of "technically correct but completely irrelevant" error is emerging as a new blind spot when AI assists in understanding complex application graphs.

An Impressive Report Card for Factual Verification

The Statewave Guide team ran hundreds of deliberately rigorous tests on their own application using the anthropic-opus-5 model. The results were striking: zero fabricated routes reached the product model, zero fabricated permissions, zero unknown topics incorrectly introduced. For prohibited capabilities, the model refused all 27 requests; for unsupported operations, it declined all 23 requests. The verifier's performance on the factual safety front was virtually flawless.

But the team then disclosed a set of sharply contrasting numbers: the model had an 89% chance of finding the correct evidence, yet only a 20% chance of turning it into a verifiable product assertion. 71% of features ended up with no verified capability description at all, and the workflow recovery rate was just 18%. One developer joked in the post: "Technically very safe. In practice, it's like a meticulously verified empty room."

What is an "application graph"? In this class of code comprehension tools, an "application graph" refers to a structured knowledge representation of the target software: abstracting elements such as routes, components, permissions, services, and data flows into nodes, and relations such as calls, dependencies, and containment into edges, forming a graph structure that the model can query. It resembles an extended version of a call graph or dependency graph in software engineering, but with broader coverage that also incorporates the UI layer and business semantics. The value of an application graph lies in pre-organizing facts scattered throughout the source code so that the AI assistant doesn't have to reason from scratch each time, but can instead retrieve from a known neighborhood. Understanding this term is key to understanding the whole article: all of Statewave Guide's experiments are essentially testing whether "the model can pick out the assertion that belongs precisely to the current feature" from the graph.

When the Truth Doesn't Belong to the Feature You Asked About

The really interesting bug surfaced on a seemingly bulletproof assertion. The model output: "Settings changes are saved by submitting the settings form." Structurally, this statement is entirely correct—the form exists, the form submits, the relation holds, every verification rule passed. The only problem: this fact holds true across the entire application, but doesn't belong to the specific feature being analyzed at that moment.

The verifier answered "does this statement have real evidence supporting it," but not "does this evidence actually belong to this feature." The distinction between these two questions forms the single most important rule of the day: Truth is not enough; context owns truth. The application graph is essentially a neighborhood—related components, sibling elements, routes, and services are naturally adjacent. If the assistant can grab any true fact from the neighborhood, it can generate technically correct but completely irrelevant answers.

Feature Scope is the intermediate layer the team introduced to fix the above blind spot. Previously, the verifier only made a binary judgment: can real evidence for a given assertion be found in the graph? With Feature Scope introduced, the graph first determines which feature is currently being analyzed and which nodes and edges correspond to that feature, and only then generates candidate assertions from within this subgraph. This is equivalent to splitting the question of "does the truth hold" into two steps: first locate it in the correct context, then judge whether the assertion itself holds. The existence of this layer directly explains why the assertion production rate later rose from 20% to 95%—the model no longer has to guess which fact belongs to which feature across the entire graph, because feature boundaries are delineated in advance by the graph.

From Model Guessing to Graph Guidance

The team realized they had previously asked the model to do too much work: finding evidence, choosing assertions, choosing operations, choosing subjects, choosing relations, and then explaining. Yet the source graph already held most of the information. So the design direction shifted from "model → fabricate assertion → verifier" to "graph → deterministic assertion opportunities → model adds meaning → verifier."

The new pipeline preserved the same safety boundaries but dramatically reduced the model's room for guessing. The results showed directly in the numbers: the verified assertion production rate jumped from about 20% to 95%, incorrect feature assertions dropped to 0, and model restraint reached 30/30. Yet one metric didn't move—45 of 63 features still ended up with no verified capability. In the author's words: "Safer, but not fuller."

Deterministic assertion opportunities refers to a design principle: rather than letting the model run free and having the verifier filter afterward, the graph proactively enumerates all "fact combinations that can be safely asserted," and the model is then tasked with giving these combinations natural-language explanations. The core of this shift is downgrading the model's role from "fact producer" to "phrasing agent." There are three benefits: first, the model no longer has the opportunity to fabricate facts that don't belong to the feature; second, the incorrect assertion rate can be driven directly to zero, because every assertion originates from a real edge in the graph; third, verification cost drops substantially. The trade-off is that coverage improvement depends on the completeness of the graph itself—if the graph lacks a capability description for some feature, the mapping stage can't generate assertions either. This is precisely the root cause of the later finding that "45 features still have no verified capabilities."

Can Safe Product Knowledge Survive Real Users?

Statewave Guide is built on Statewave, an open-source, self-hosted AI agent memory runtime that emphasizes governance and provenance rather than mere retrieval, licensed under Apache 2.0. After achieving extremely high verification safety, the team raised a question they should have asked earlier: can these mechanisms actually help humans?

This leads to the next core question of the series: can technically safe product knowledge survive contact with real users? For developers building AI coding assistants, code comprehension tools, or enterprise knowledge bases, the answer to this question may matter more than the verification pass rate. The real challenge may not be getting the model to avoid saying wrong things, but getting it to say things that are genuinely useful in the current context.

One-line Summary

AI factual correctness isn't relevance: Statewave Guide finds a significant gap between verification pass rates and feature coverage.

Paragraph Summary

The Statewave Guide project reveals a new blind spot in AI-assisted code comprehension: factually correct assertions can still be completely irrelevant to the feature the user is asking about. Experiments show that the model finds correct evidence 89% of the time, but converts it into verifiable product assertions only 20% of the time, with 71% of features having no verified capability description at all. The root cause is that the verifier only judges "does the fact hold," not "does the fact belong to the current feature." The team therefore introduced a Feature Scope layer and a graph-guided deterministic assertion mechanism, shifting focus from the model fabricating assertions to the graph proactively enumerating opportunities, raising the verified assertion production rate from 20% to 95% and driving incorrect feature assertions to 0. But coverage remains limited—45 of 63 features still have no verified capabilities, and the safety improvement didn't come with a matching increase in coverage. The author thus raises the core question of the next phase: can technically safe product knowledge survive in front of real users?