Let AI Audit Its Own Docs: An Open-Source Toolkit for Verifying CLAUDE.md Accuracy

The developer behind the 29k-star RAG Techniques repository, Nir Diamant, has open-sourced a toolkit designed to address a problem most people overlook: how much of the project documentation an AI Agent reads is actually true?

A Sobering Experiment

It started simply enough. The author assumed his repository documentation was clear — until he decided to let an AI Agent put that to the test.

Before letting Claude read a single line of code, he posed five questions about his own project: What tools are installed? What rules does the linter enforce? Where should new files go? Crucially, he asked Claude to label each answer with a confidence rating — SURE, GUESS, or WOULD-HAVE-TO-LOOK.

The result was telling. In one answer marked SURE, Claude named two tools that weren't installed anywhere in the codebase. They existed only in the author's spec document — as planned additions. Claude had treated that plan as the actual tech stack.

This detail exposes a subtle trap in AI-assisted development: intentions written in documentation are easily misread as implemented reality.

Code as the Single Source of Truth

The author then had the Agent do two things in reverse: generate a documentation layer from the code itself, and use the code to fact-check the existing docs.

The findings were equally sobering — five claims in the README and spec were contradicted by the code. Among them: a scheduler that was never chosen in the spec document but was actually doing the work. The architecture described in the docs and the code actually running had quietly diverged.

This is the toolkit's core philosophy: documentation shouldn't merely sound confident — it should be verifiable against running code.

This idea has deep roots in software engineering. "Code as the single source of truth" is the counterpoint to Documentation-Driven Development and a foundational assumption in many modern DevOps practices. In traditional development, docs and code begin to diverge the moment a change is made — code gets updated while documentation relies on human discipline to follow suit. This drift is tolerable in small projects but gets amplified when AI Agents are consuming the docs. Unlike humans, Agents can't use common sense to bridge the gap between documentation and reality; they can only work with the text they're given. This is why some mature engineering practices now advocate for "executable documentation" — machine-verifiable formats like test cases and OpenAPI schemas — to replace or supplement natural language descriptions and prevent documentation rot at the source.

How to Use the Tool

The workflow is intentionally minimal. You simply paste a single instruction to an Agent already working in your repository:

Clone https://github.com/NirDiamant/Agentic_Engineering into a temp folder, read its RUN.md, and follow it on this repository.

About fifteen minutes later, you get three things:

• A docs/ layer generated from the actual code
• A CLAUDE.md (or AGENTS.md) in the root directory
• A card populated with your project's real data

The most important design decision is how uncertainty is handled: anything that can't be confirmed against running code is flagged <!-- unconfirmed --> rather than presented as fact. The tool doesn't delete anything and pauses once to ask for input. This conservative approach reduces the risk of automated documentation rewrites going wrong.

Comparison with Claude Code /init

To check whether he'd reinvented the wheel, the author ran Claude Code's built-in /init command on the same repository.

His assessment was candid: /init did a solid job and even caught a bug he'd missed himself. His first database migration script created a table; his second script created the same table again with if not exists — running both migrations in sequence would break the newer path. This is exactly the kind of issue that slips through manual review.

This comparative stance is itself instructive: open-source tools and official capabilities aren't zero-sum competitors — they can complement each other and cover different blind spots.

/init is a slash command built into Claude Code (Anthropic's developer-facing CLI AI coding tool) that scans the current codebase and auto-generates a CLAUDE.md file — essentially a "project brief" for Claude, covering build processes, code conventions, common commands, and other context. The idea behind CLAUDE.md is that Claude can read this file at the start of a session and immediately understand the project without needing to re-learn it each time. The distinction from this toolkit is that /init focuses on generating a context file for Claude's own use, while this toolkit places additional emphasis on cross-verifying existing documentation and explicitly flagging uncertain content — the two tools have somewhat different target audiences and priorities.

What It Can't Do

The author is unusually honest about the tool's limitations — a refreshing quality in open-source project announcements.

First, there's no mechanism to enforce the generated files — they're documentation, not constraints. Second, and more fundamentally: the final review is essentially one model checking another model's work. It cannot actually tell you whether the documentation is correct.

There's also a subtler issue around environment differences. In Claude Code, sub-agents enable a degree of independent cross-verification, but in Codex or Cursor there are no sub-agents — the session that writes the files and the session that scores them are the same one. Writing and grading your own work lacks independence. In these environments, the reliability of verification is inherently reduced.

Sub-agents are a task decomposition pattern in Agentic engineering: a primary Agent breaks down a complex task and spawns independent sub-Agents to handle specific subtasks, then aggregates the results. The key advantage of this architecture is isolation — the Agent writing documentation and the Agent verifying it run in separate contexts or even separate sessions, theoretically reducing confirmation bias. Claude Code natively supports this kind of division of labor, while Cursor and Codex currently use interaction models closer to a single continuous session, with no built-in sub-agent scheduling. This means that in the latter two environments, the same model instance scores its own output immediately after producing it — a fundamentally different level of independence compared to the cross-verification that sub-agents enable in Claude Code.

Implications for AI Engineering Practice

The toolkit itself may be modest, but it points to an increasingly important question in Agentic engineering: when we let AI understand and modify code based on documentation, the trustworthiness of that documentation directly determines the reliability of the Agent's behavior.

Treating "intent" as "reality," letting a stale README mislead Agent decisions — these problems will be amplified as AI takes on more work within codebases. Confidence labeling, reverse-verification from code, and honest flagging of uncertain content are practices worth building into everyday AI-assisted development workflows.

The project is licensed under Apache 2.0 and completely free. For teams making heavy use of Claude Code, Cursor, or Codex, spending fifteen minutes auditing your documentation layer might surface some unexpected findings.