LLM Wiki Setup Guide: From Knowledge Ingestion to Agent Collaboration

When knowledge is scattered across dozens of vendor docs, research PDFs, and loose notes, every conversation forces you to restate the background from scratch. That repeated drain is becoming a hidden cost of AI-assisted work. Andrej Karpathy's LLM Wiki model offers a structured answer: organize knowledge into Markdown pages with YAML metadata, and let humans and AI agents maintain and query it together.

One developer put this approach into practice at work and adapted it to a project called Cereja Flamejante. Here are the key decisions behind the system, from design to implementation.

Why You Need an LLM Wiki Instead of Ordinary Notes

The problem with ordinary notes isn't storage — it's the lack of an actionable structure. The core distinction of an LLM Wiki is a three-layer design: original sources are fully preserved, related pages are linked to each other, and a guide document tells agents how to maintain the whole knowledge base.

The model Karpathy describes has three components: retained raw files, interrelated note pages, and a specification document that instructs the AI agent on maintaining the wiki. The developer adapted this further, using local Obsidian sync to read notes and browse link relationships. The graph view reveals connections between pieces of knowledge, but the author warns specifically: a pretty connection graph is not the same as a claim backed by sufficient evidence. To verify information, you must go back to the original source.

There's a key distinction here: persistent knowledge and agent context are two different things. A knowledge base may have hundreds of notes, but a specific task needs only a few of them. RAG (Retrieval-Augmented Generation) handles retrieving relevant snippets from the knowledge base, while knowledge organization itself serves an entirely different function.

RAG (Retrieval-Augmented Generation) is a technical paradigm that combines an external knowledge base with a generative model: before the model generates an answer, it first uses vector search or keyword matching to find the most relevant text snippets from the knowledge base for the question, then feeds those snippets as context into the model. Its core value is letting the model answer based on traceable external evidence rather than relying solely on parametric memory fixed during training, thereby reducing hallucinations and supporting knowledge updates. A typical RAG pipeline involves splitting documents, generating embeddings, building an index, retrieving candidates, reranking, and assembling context. Its relationship to this wiki is complementary: the wiki organizes knowledge into a stable, reviewable, source-traceable structure, while RAG recalls a small number of snippets on demand for specific tasks. Only by understanding this division of labor can you see why a carefully maintained knowledge base does not become redundant just because RAG exists.

How the Wiki Embeds into Agent Workflows

In Cereja's architecture, Núcleo records knowledge, evidence, interpretations, and arguments; the Editorial Engine prepares content for publishing; and the Agentic Factory maintains agent roles and shared controls across systems. The wiki's role in this architecture is to make knowledge queryable.

Guides direct specific work: they describe the process, the sequence, and the information a task requires. For example, if an agent is to prepare an article about a study, it needs the sources, limitations, and editorial instructions. The agent can propose a synthesized judgment, but approving that judgment as Cereja's official position and authorizing publication are two separate decisions.

This is the role of the agent layer: organizing queries and tool use during execution. It needs to know two things at once — where to find knowledge and what actions it can take.

In this workflow, some steps are still done manually.

Ingestion Starts with a Command, Not Automatic Triggering

In the developer's original work wiki, dropping a document into the raw/ directory does not automatically trigger ingestion. Someone needs to actively request processing, or handle it at the end of a task.

The system uses a check to find files not referenced by any note. If a source is intentionally discarded, the reason is recorded. This way the file no longer appears as pending, but it doesn't disappear from history either. There's no watcher and no scheduled task — because folders don't change every minute, and this workflow is sufficient in practice.

In Cereja, the same triggering method is used: a command. The implementation order is to build the source trace record first, then extraction and synthesis.

raw/                       # Original local files
sources/                   # Source traces and reading notes
tools/ingest_source.py      # Ingestion command
examples/wiki-ingestion/  # Cereja's own examples

The command computes the file's SHA-256 hash. This hash identifies the byte content of the file and is incorporated into the note filename. If the note already exists, the command returns its path and preserves existing annotations.

 digest = hashlib.sha256(path.read_bytes()).hexdigest()
 destination = target_dir / f"source-{digest}.md"
 
 if destination.exists():
     return destination, False

The work wiki uses the normalized source address (recorded in resource) as the identity. In Cereja, these two functions are separated: the hash identifies content, the citation records the source. This can detect different byte contents but doesn't solve all duplication issues — two PDFs might represent the same study. Version updates also require linking different versions.

SHA-256 is a cryptographic hash function in the SHA-2 family that maps an input of any length to a fixed 256-bit (64 hexadecimal character) digest. It has three properties important for this scenario: determinism (the same byte content always yields the same hash), the avalanche effect (changing even one byte of input drastically changes the output), and collision resistance (it's extremely hard to find two different inputs that yield the same digest). Therefore, using the SHA-256 of file content as the note filename naturally implements "content addressing": ingesting the same file repeatedly hits the existing note, while different content automatically generates a new note. Note that the hash identifies the byte level; two different PDFs describing the same study will produce different hashes, so the system still needs a citation field to record source identity. Only together can the two detect content differences and handle semantic duplication.

Source Tracing: Every Note Must State Its Origin

Cereja notes are created in draft status, recording the title, declared author, original file path, hash value, URL when available, declared rights status, and ingestion date. The command marks unknown fields as unknown.

You can run the implementation's own example with the following command:

 python tools/ingest_source.py examples/wiki-ingestion/source-demo.md \
   --title "Cereja Collection Example" \
   --author "Cereja Flamejante" \
   --rights "Own demo material" \
   --public-note

The --public-note option confirms that the filename and metadata can appear publicly, but it does not grant any rights to the document itself. Raw files in raw/ are excluded from Git by default. You must check a PDF's license before publishing it. When redistribution isn't allowed, you can keep the citation and your own original analysis while respecting the source's rights.

Synthesis Happens After Ingestion

In the work wiki, PDF text extraction uses Poppler's pdftotext. The knowledge base has no OCR capability. When a document contains image-based pages, the model can read them directly as images.

The LLM writes synthesized content and annotates page numbers. Notes record statements from the document separately from our own interpretations and reservations. The model used and token consumption are also recorded. Concept and entity notes point to source notes.

In Cereja, the current command only creates the source trace record. These fields make the current state clear at a glance:

 type: "source"
 status: "draft"
 certainty: "unverified"
 published: "unknown"
 extraction_status: "not_attempted"

In this workflow, document extraction and summarization are not yet implemented. After reading, you need to record the method, population or scope, limitations, and the original passages supporting each claim. Cereja's evidence schema guides this review.

A generated synthesis only becomes canonical knowledge (a system-recognized reference) after human promotion. And "approved by Kell" is not the same as "written by Kell" — the authorship trace is always preserved.

Check Mechanisms: Who Reviews What Gets Missed

The check in the work wiki validates metadata, links, note length, and presence in the index, marks unprocessed sources, and verifies whether derived notes retain the source's constraints.

Notes start as draft. Someone records reading status in review_stage. Promotion to stable requires a human owner; the check rejects attributing stable notes to a model. Whoever opens a PR does not approve their own PR.

These validations help maintain the knowledge base, but they don't read the research for us. A link may work fine but point to an inappropriate source. A human is still needed to assess what actually supports a claim.

In Cereja, checks for coverage, rights, and promotion still need adaptation and testing. At the same time, the status definitions Núcleo already uses must be respected. Copying names without verification only creates new confusion.

On-Demand Querying: Don't Cram the Whole Knowledge Base into Context

In Núcleo, the workflow starts from the task, locates applicable rules, and consults the needed materials. This set enters the agent context, while the rest of the knowledge base remains queryable.

Anthropic describes this on-demand querying as a context organization strategy, and also points out a cost: the agent needs to explore the knowledge base during execution.

When preparing content for a newsletter, a task might require comparing two studies. The agent receives the sources and their limitations, along with writing instructions. The conclusion it proposes still needs review.

More documents don't solve every failure. Sometimes what's missing is a piece of information; sometimes it's a constraint imposed by a tool or validation of a result. Before adding another folder, figure out which piece is actually missing.

The Portability Issue OKF Brings

Google Cloud's Open Knowledge Format (OKF) advocates representing knowledge in Markdown, organized with YAML metadata and links between concepts.

This is worth noting because it separates the knowledge format from the tools that produce and query knowledge. A knowledge base can be shared across tools without depending on a single interface.

OKF v0.1 is still a preliminary proposal. Using Markdown and metadata doesn't automatically make a knowledge base compatible with it — you need to check the spec item by item.

YAML (YAML Ain't Markup Language) is a serialization format that expresses hierarchy through indentation and fields through key-value pairs, commonly used in configuration files and metadata headers of data files. In this wiki's notes, the region wrapped by --- at the top of the file is YAML front matter, used to declare structured fields like type, status, certainty, and published. Compared to Markdown body text, YAML metadata is better suited for programmatic parsing and validation, so the check mechanism can read these fields to determine note status, link integrity, and promotion eligibility. YAML is sensitive to indentation and the space after a colon. A common practice is to keep field names and value styles consistent, and let automation tools handle writing rather than writing by hand, to reduce parsing failures caused by formatting errors.

First Test and Next Steps

The developer ran the command with a self-authored Markdown demo file. The first call created the note. The second returned the same note without overwriting it.

The test validated byte preservation, preservation of manual annotations, re-ingestion, content change, path boundaries, and public metadata confirmation. The implementation is in PR #34, and two new tests pass. The full test suite showed a pre-existing Flame failure related to Windows path separators.

These results confirm the command's specific behavior but do not yet measure time savings or improvements in curation quality. PDF extraction, reviewable synthesis, and coverage checks are next. OCR and folder monitoring are left for when actual use proves the need.

The author's goal is to use this knowledge base to gather research, track issues, and prepare Cereja's content curation. Starting with source tracing comes from the need to be able to go back to the document and understand what has been verified. The next test is using these notes in a real newsletter topic.

In your knowledge base, how do you distinguish an AI-generated note from information that has been human-reviewed?

One-line Summary

LLM Wiki uses command-driven ingestion, hash tracing, and human review to make the knowledge base queryable and trustworthy.

Paragraph Summary

This article introduces an LLM Wiki approach proposed by Andrej Karpathy and implemented in the Cereja Flamejante project. It organizes knowledge into Markdown pages with YAML metadata, including a three-layer structure: retained original sources, interrelated note pages, and a specification document guiding AI agents to maintain the knowledge base, with the core distinction being actionability and traceability. Ingestion is actively triggered by a command rather than a watcher, using a SHA-256 hash to identify file content and serve as the note filename, achieving deduplication and source tracing. The article emphasizes that persistent knowledge and agent context are two different things, and on-demand querying avoids cramming the whole knowledge base into the model context. Notes must be human-promoted from draft to stable, and the check rejects attributing stable notes to a model. OKF v0.1 provides a format specification for cross-tool sharing but still requires item-by-item compatibility checking.