KLM Model: Making AI Output Evidence Visible, But Don't Rush to Trust It

AI answers that look correct don't necessarily mean they're truly reliable. Reviewers need to be able to examine the documentary basis behind an answer. Recently, an open specification called the Knowledge Layer Model (KLM) was proposed, attempting to establish an implementation-agnostic standard for inference records, making evidence, epistemic assessments, and unknowns all visible. At the same time, the accompanying reference implementation SEDIM has also disclosed a series of internal experimental results—including both impressive data and a large number of negative results. These findings remind us: evidence visibility itself does not equal correctness.

What Is KLM: Establishing an "Evidence Layer" for Inference Records

KLM (Knowledge Layer Model) is an open, implementation-neutral specification for defining inference records. These records contain versioned epistemic assessments, evidence relations, and explicit unknowns. Simply put, it attempts to answer a question: when an AI gives an answer, can we trace what information it's based on, at what confidence level, and where uncertainty exists?

SEDIM is the knowledge and evidence architecture of Nage AI OS, existing as the reference implementation of KLM. The currently cited version is KLM v0.3 (repository tag v0.3.0). It should be emphasized that these empirical materials come from independent internal experiments, not end-to-end validation. In other words, KLM provides a framework, but its actual effectiveness still requires more independent replication.

Routing Diagnostics: 91.3% Classification Accuracy and Its Limitations

In a historical platform-domain routing benchmark test, the system completed 137/150 correct classifications across 15 categories, with an accuracy of 91.3% and a nominal 95% Wilson interval of 85.7-94.9%. This number looks good, but the authors candidly acknowledge several key limitations: the numerator was recovered from a rounded summary, and the attribution of historical samples was unverified. This means the 91.3% figure itself carries some uncertainty and cannot be directly cited as a precise performance metric.

More noteworthy is another test: in the corrected dual-source test, the native model weight path configuration achieved 0/15 successful answers. This is a complete failure case. It demonstrates that under certain configurations, even with a framework like KLM, the model's own performance may be completely unusable. The value of routing diagnostics lies precisely in exposing such problems, not concealing them.

Wilson interval is a confidence interval estimation method for binomial distribution proportion parameters, particularly suitable for small samples or extreme proportions (close to 0 or 1). Unlike the traditional Wald interval, the Wilson interval introduces a quadratic correction term, avoiding the problem of unreasonable interval widths or even negative values when sample sizes are insufficient. In the 137/150 classification accuracy reported in the article, the nominal 95% Wilson interval is 85.7%–94.9%, meaning that if experiments of the same scale were repeated, the true accuracy would fall within this range with 95% probability. However, it should be noted that the validity of this interval depends on sample independence and representativeness; the article has already pointed out that the numerator came from a rounded summary and sample attribution was unverified, so this interval can only serve as a reference and cannot be regarded as a strict statistical guarantee.

Negative Results: AUC 0.008 and Deficiencies in the Scorer

The most striking aspect of KLM-related experiments is a series of negative results. Vocabulary diagnostics and scope diagnostics both achieved an AUC of 0.008 on "silent versus contradiction" samples, far below the registered minimum requirement of 0.65. An AUC of 0.008 means the diagnostic signal is almost completely unable to distinguish between the two classes of samples—a level close to random or even worse. The authors did not avoid this result but reported it faithfully.

Another capacity growth comparison could not establish reliable effects because the legacy scorer had defects and the original outputs were missing. In legacy packing, 118/221 selected cases lost content containing gold-standard text; the fixed packing retained this text in 221/221 cases. But the authors explicitly state that this measures text presence, not answer quality. Text being present doesn't mean the answer is correct.

AUC (Area Under the ROC Curve) is a commonly used metric for evaluating the discriminative ability of binary diagnostic signals, with values typically ranging from 0 to 1. AUC=0.5 indicates the diagnostic signal is no different from random guessing; the closer AUC is to 1, the better the signal can correctly distinguish between positive and negative samples. The article mentions a registered minimum requirement of 0.65, meaning the diagnostic signal should have at least moderate discriminative ability. An AUC=0.008 means the diagnostic signal is almost completely reversed or ineffective—it not only fails to distinguish between "silent" and "contradiction" samples but performs far worse than random guessing. Such extreme low values typically indicate serious mismatch between diagnostic features and target labels, data leakage, or scoring logic defects, requiring fundamental investigation rather than simple parameter tuning.

Validators and Unfinished Assessments

Python and TypeScript validators are reported to agree on six checks per set of synthetic fixtures; however, an independent production disclosure was not accepted. Recovery, re-derivation, human scoring standard support, boundary isolation, and bounded response assessment remain at the proposal stage.

All these findings come from internal sources and are self-reported by the authors, with historical observations and later replications explicitly distinguished. The authors repeatedly emphasize: evidence visibility does not establish correctness, causal dependence, safety, or regulatory compliance. Independent replication and a complete public audit package remain future work.

Synthetic fixture refers to a pre-built data set used in software testing to simulate real input scenarios, typically constructed manually to cover specific boundary conditions or exceptional paths. In KLM's validator tests, Python and TypeScript validators agreed on six checks per set of synthetic fixtures, indicating that both implementations produced identical validation results for the same set of pre-built inputs. However, the limitation of synthetic fixtures is that their coverage is predetermined by the designers and cannot fully represent the complexity and unpredictability of production environments. The article specifically notes that "an independent production disclosure was not accepted," suggesting that consistency in synthetic testing cannot be directly generalized to real production data, and the actual reliability of validators still needs to be tested under conditions closer to actual deployment.

Why This Matters

The value of KLM lies not in what it proves, but in its attempt to establish a culture of auditable AI output. Most current AI systems only provide answers, not the "evidence chain" behind those answers. KLM requires recording the version of epistemic assessments, evidence relations, and unknowns, providing a foundational structure for review and accountability.

But negative results are equally important. AUC 0.008, 0/15 successful answers, scorer defects—these numbers demonstrate that there is still a long way to go from "evidence visible" to "evidence reliable." An honest framework should report both successes and failures, and KLM's experimental disclosure sets an example in this regard. For practitioners concerned with AI interpretability and auditing, this specification is worth tracking, but it should not be prematurely regarded as a mature solution.

Conclusion: Visibility Is Only the Starting Point

KLM and SEDIM propose an approach to make AI inference records inspectable, but the large number of negative results in internal experiments reminds us: the evidence layer itself needs to be validated. Routing diagnostics can expose failures, but diagnostic metrics themselves can also fail (AUC 0.008 is an example). Until independent replication and public audit packages are completed, empirical claims about KLM should be treated with caution. Evidence visibility is a necessary first step, but far from the endpoint.