Kyle Kingsbury on How LLMs Are Eroding Trust in Software Documentation

An Industry Reflection Sparked by a Single Quote

Simon Willison's blog recently quoted a comment from Jepsen's analysis of ParticleDB 1.1.4, written by Kyle Kingsbury, a well-known engineer in the field of distributed systems testing. Although brief, this passage highlights a growing industry anxiety: as large language models (LLMs) are widely used to generate code and documentation, the traditional signals we use to judge software maturity are becoming unreliable.

Kingsbury's core point is that the use of LLMs erodes trust in documentation, software, and the organizations that produce that software. He further predicts that engineers may begin to view any new software project with skepticism, because the signals once regarded as markers of a mature system—thorough documentation, clear changelogs, active community maintenance—are now easier to fake.

This passage is worth discussing not because it disparages AI, but because it raises a very concrete engineering management question: in the LLM era, what should maintainers do to ensure the correctness of software and documentation? And once that work is done, how do they prove its quality to users?

The Cost of Faking Trust Signals Is Falling

In both open-source and commercial software, users often rely on indirect signals to judge whether a project is reliable. Whether documentation is complete, release notes are detailed, issues are answered promptly, and code comments are clear—these surface-level indicators, which previously required substantial human effort to maintain, can now be generated at low cost and at scale with LLMs.

The problem is that such generated documentation can look highly professional while not necessarily matching the actual behavior of the code. A project can use an LLM to generate a full site of polished API documentation, but the parameter descriptions, boundary conditions, and error code meanings may diverge from the implementation. For users who do not read the source code deeply, such discrepancies are extremely hard to detect, and once triggered in production, the cost is often high.

This is exactly what Kingsbury means when he says that "the markers of a mature system are now easier to fake." The cost of generating trust signals has fallen, but the cost of verification has not fallen in tandem—and the gap between the two is where the risk lies.

The "surface coherence" trap of LLM-generated content: Large language models excel at generating fluent, well-structured text with accurate terminology. In documentation contexts, this ability creates a kind of "surface coherence." Readers find it hard to notice logical leaps between paragraphs or details that do not match the implementation, because the text itself reads flawlessly. Unlike the stiff output of traditional machine translation or template generation, LLM output can be almost indistinguishable from genuine writing. As a result, documentation reviewers need to shift from asking "does it read smoothly?" to asking "does it strictly correspond to code behavior?"—for example, by using doctests, comparing interface descriptions against OpenAPI specifications, and automatically validating parameter defaults, thereby separating consistency checks from manual reading.

The Maintainer's Dilemma: Doing It Right, and Then Proving It

Kingsbury raises two levels of problems in the quote. The first is practical: how maintainers can ensure the correctness of software and documentation. This requires teams to establish corresponding verification processes when introducing LLM-assisted generation—test coverage, consistency checks between documentation and code, and manual review of critical passages.

The second is more difficult: even if maintainers do this work, how do they prove it to external users? In the past, detailed documentation was itself the proof; now, detailed documentation may simply be the product of an LLM. The mechanisms for building trust need new anchors.

Possible anchors include reproducible test results, public benchmarks, third-party audit reports, and independent analyses such as Jepsen's. The Jepsen team, where Kingsbury works, does exactly this kind of work: verifying the consistency guarantees of distributed databases through rigorous fault-injection testing. The value of this kind of verification is actually amplified in the LLM era, because it is difficult to fake at low cost.

Verifiability and trust anchors: In software engineering, verifiability means that a system or its artifacts can be independently checked by a third party in a repeatable way against their claimed properties. The core of the trust crisis in the LLM era is the asymmetry in verification cost between "claims" and "facts." Therefore, trust anchors need two characteristics: first, they are difficult to fake at low cost; second, the verification process does not depend on the maintainer's self-report. Common strong anchors include hashes of reproducible builds, publicly available test coverage reports in continuous integration, outputs of formal verification or model checking, and signed third-party audit reports. Weak anchors include the number of documentation pages, update frequency, and issue response speed—exactly the signals LLMs are best at mass-producing. Maintainers should proactively embed strong anchors into the release process, for example by attaching the hash of the CI log and an execution summary of the test suite to each release.

Practical Impact on Developers' Daily Decisions

The direct takeaway of this comment for ordinary developers is: when evaluating a new dependency or tool, do not judge solely by how complete its documentation is. Good documentation no longer equals good project maintenance. A more reliable approach is to check the actual coverage of the test suite, whether there is independent third-party verification, and whether the community has real use cases and issue feedback.

For teams maintaining their own projects, this means placing greater emphasis on "verifiable correctness" rather than "looking professional." Investing effort in tests, CI, and reproducible builds builds long-term trust more effectively than generating more documentation pages. LLMs can accelerate documentation production, but the credibility of documentation must ultimately rest on the code and tests themselves.

Kingsbury's passage does not provide a complete answer, but it accurately describes the problem. As LLMs become widespread, the software industry's trust infrastructure needs to be rebuilt, and this requires the participation of maintainers, users, and independent verification organizations alike.

"Signal decay" in the software supply chain and dependency evaluation: In dependency evaluation, developers face a supply chain characterized by information asymmetry. Traditionally, signals such as documentation completeness, download counts, and star counts have been used as proxy metrics, but LLMs can easily amplify these proxy metrics without any real investment in quality. This leads to signal decay—a decline in the correlation between metrics and true quality. The response strategy is to shift to "behavioral evidence": check whether a project has published fault-injection tests, long-running fuzzing results, and records of responses to known CVEs. In addition, pay attention to whether the project uses a traceable code review process (such as requiring every PR to include test changes), because such processes are difficult for LLMs to fully automate around.

One-line Summary

LLMs lower the cost of faking maturity signals, and software trust mechanisms urgently need to be rebuilt.

Paragraph Summary

Starting from Kyle Kingsbury's comment, the article points out that the spread of LLMs is eroding the credibility of traditional software maturity signals. Trust markers such as documentation, changelogs, and community activity—which previously required substantial human effort to maintain—can now be generated at low cost and at scale, and may look professional while not matching the actual behavior of the code. Maintainers face a dual challenge: ensuring the correctness of LLM-assisted output and proving that correctness to users. The article argues that hard-to-fake anchors such as independent third-party verification (e.g., Jepsen's fault-injection testing), reproducible test results, and third-party audits are increasing in value. For developers, evaluating dependencies should focus more on test coverage and independent verification than on documentation completeness; for maintainers, effort should go into verifiable correctness rather than surface professionalism.