LLM Wiki vs RAG: Real-World Test Shows Replacing Retrieval with a Wiki Is Actually Slower

Treating an LLM wiki as just another way to do RAG is a common misconception. Vellum Labs ran a controlled experiment on a real codebase, and the conclusion is somewhat counterintuitive: on small codebases, using a wiki as a retrieval substitute is slower and more expensive than not retrieving at all, yet the number of correct answers is the same.

Three Concepts, Don't Confuse Them

RAG (Retrieval-Augmented Generation) uses a retriever at query time to find the most relevant chunks from a document library, stuffs them into the prompt, and the model answers based on those chunks. The document library is a copy of the source files, chopped up and left in place; the next query starts from the same library with no write-backs.

Agentic retrieval is the variant actually used by today's coding agents: no embedding index, the agent repeatedly runs grep, opens files, and reads until it can answer. The retriever is the model itself with a shell tool. In the experiment, the "no wiki" side refers to this mode; the author did not run a vector-database version of RAG.

An LLM wiki is a markdown folder that the agent itself writes to and maintains: one page per concept, decision, or module, each claim annotated with its source (file:line, changelog version, or dated conversation), pages cross-referenced with [[wikilinks]], plus an index the agent reads first. Source files sit alongside, unchanged. When a page can't answer a question, the agent goes to the source files and writes back what it learned into the page so the next question doesn't have to re-read the source.

In one sentence: RAG retrieves what has been written down; a wiki also records what was decided. Every other difference follows from this premise.

Trade-offs between agentic retrieval and vector RAG: Vector RAG relies on an embedding index and similarity ranking, which suits scenarios with large corpora and fuzzy query semantics — but the index itself must be built and maintained, and chunk boundaries lose cross-file context. Agentic retrieval (grep, open files, read layer by layer) builds no index; the model decides what to read and when to stop. Its strengths are semantic accuracy and traceable citations; its cost is re-traversing files on every query, accumulating linearly with the number of questions. On small codebases, files are few and jumps are shallow, so the traversal cost of agentic retrieval is low — cheaper than maintaining an index. But when a repository reaches hundreds of thousands of lines, an agent may need to read dozens of files before converging, and per-question cost rises sharply; only then does the amortized value of a pre-built index or wiki show up. Understanding this is the key to understanding why, at the 1,176-line scale in this article, "no wiki" wins.

Experimental Setup and Results

The test subject is http://pallets/itsdangerous — the signing library behind Flask sessions: 8 modules, 1,176 lines of Python, 292 lines of CHANGES.rst, and 7 pages of docs. Using Claude Code 2.1.286's default model, each run executed in headless mode via claude -p in a fresh session, with cost taken from the CLI's total_cost_usd.

The four questions are ones developers really ask when integrating the library: whether old tokens remain verifiable after key rotation, what the salt parameter actually does, why JWS support was removed, and how timestamps are encoded in TimestampSigner and when the format changed.

No Wiki: The Agent Reads Source Directly

Question	Time	Turns	Cost	Correct
Key rotation	33 s	6	$0.62	Yes
salt	28 s	3	$0.45	Yes
JWS removal	28 s	4	$0.38	Yes
Timestamp format	35 s	5	$0.47	Yes
Total	2 min 4 s	18	$1.93	4 / 4

Every answer was checked against the cited files; all four were correct, and the cited lines actually exist.

Building the Wiki

Initialization plus construction took 7 min 51 s and cost $4.23, producing 17 pages, 6,331 words, 86 [[links]] (0 broken, 0 orphan pages), and 110 file:line references (0 pointing to non-existent lines). The author notes: the same prompt on the same commit produced 21 pages, 8,441 words, and $4.08 two days earlier — construction is not deterministic; if you treat page count as a quality metric, run the checker rather than trusting memory.

[[wikilinks]] and citation audits: wikilinks are a bidirectional page-linking syntax that originated in early wiki systems and has been carried forward by tools like Obsidian and Logseq. Their value isn't just navigation — it's auditability: if page A links to a non-existent page B, you get a "broken link"; if a page has no pages linking into it, it's an "orphan page." The article's figures of 17 pages, 86 links, 0 broken links, and 0 orphan pages are exactly how structural health of a wiki is measured. Alongside this is the file:line citation format — source-file coordinates precise to the line number — which lets every claim be regression-verified programmatically; the article deliberately checks whether all 110 references point to real lines. This machine-verifiable citation chain is a key engineering property of a wiki compared with ordinary notes, and is also the direct reason queries must return to the source to verify, making them slower.

Answering the Same Four Questions via the Wiki

Question	Time	Turns	Cost	Action
Key rotation	57 s	6	$0.82	Page lacked per-key verification cost; read source and wrote back to two pages
salt	66 s	6	$0.64	Answered from wiki, confirmed against docs and source
JWS removal	25 s	5	$0.50	Answered from decision page, no changes
Timestamp format	43 s	6	$0.63	Answered from decision page, no changes
Total	3 min 11 s	23	$2.59	4 / 4 correct
Re-ask Q1 after write-back	28 s	5	$0.59	Answered from wiki, no changes

What the Numbers Say

On a 1,200-line codebase, the wiki is worse than RAG-by-not-doing-RAG. In the first round, it's 55% slower and 34% more expensive per question. Even after write-back, repeating a question costs about the same as a cold read of the source (28 s vs 33 s, $0.59 vs $0.62), because the query command still verifies citations against source files before answering. The savings from retrieval can never recoup the $4.23 build cost. At this library's scale, there is no break-even point.

The author honestly notes two unmeasured counter-factors: at 1,200 lines the whole library fits in the context window, whereas on a 200,000-line, 50-file repo "read the source directly" gets more expensive per question while the wiki doesn't, so a crossover should exist — but its location is unknown. Also, in headless runs no one waits for the result, and wiki answers are easier to read because they come framed by decision pages.

Why small repos have no break-even point: The wiki's cost structure is "one-time build + verification on every query," whereas reading source directly is "pure traversal per query." Let C be the build cost, v the per-question verification cost, and r the per-question cold-read cost. Only when v is clearly less than r and there are enough questions can the cumulative savings (r − v) × n cover C. In this experiment v and r are nearly equal ($0.59 vs $0.62) — the gap gets eaten by citation verification — so (r − v) ≈ 0, and no number of questions pays it back. The crossover only appears once the repo is large enough that r inflates with the question volume while wiki pages reliably supply answers and reduce return trips, pushing v down. The author marks the crossover's location as unknown precisely because r's growth curve depends on repo structure and question distribution and can't be extrapolated from a single experiment.

The Fifth Question: Where the Wiki Is the Only Option

The real watershed comes with a decision that arose after a code review — one that's not in src/, not in CHANGES.rst, not in any doc. The author simulates a team deciding during review: keep at most three keys in the app's key list, discarding the oldest every 30 days, because the wiki showed that rejected tokens require one MAC per key. They rejected two alternatives: an unbounded list and a hard single-key switchover.

Side	Time	Cost	Answer
No wiki	53 s	$0.51	"The source can't answer the first two parts," correctly explaining that the library leaves key count and cadence to the caller
capture	34 s	$0.57	Added the policy, two rejected alternatives, and cost reasoning to the decision page; also derived a conclusion the author hadn't stated: max_age should stay within 60 days
wiki query	22 s	$0.47	Three keys, discard every 30 days, the reasoning, two rejected alternatives, the 60-day inference — all cited

This isn't a performance difference — it's the difference between being able to answer and being unable to answer. RAG and agentic retrieval can only retrieve what someone has written down, and review decisions never get written down unless the tool turns it into a one-liner.

When to Use Which

Use RAG or plain agentic retrieval: when the answer is in the source and the source is stable (API docs, a library's own code, specs); when the corpus is large and read-heavy, so an index beats repeated reading; when no one needs to change the knowledge, just to find it.

Use an LLM wiki: when knowledge lives partly in source files and partly in people's heads, reviews, incidents, and chat logs; when you ask an agent the same project questions across multiple sessions and want the second to start where the first left off; when you need an audit chain — every claim cited, every change logged in log.md, everything in git.

Use both: with a large corpus, the wiki handles decisions and synthesis while retrieval handles raw material. The wiki's raw/ directory is exactly the document library RAG would index; the two don't compete for source files.

Don't use a wiki as a retrieval cache. The experiment above is the reason: if all you want is faster answers, the wiki adds build cost and verification cost without giving you anything you didn't already have.

A Few Unexpected Findings

Query time goes into verification. Three of the four wiki answers said "verified against CHANGES.rst" or "confirmed against signer.py" even though the pages already had the answer. For a tool that sells itself on citations, this is correct behavior — and the reason it will always be more expensive than reading source on small repos. If you want speed, you need a command that trusts the pages, and you need to know that's what you want.

capture chose "volatile" over "pinned." The tool lets the agent decide whether to copy key lines of a conversation into raw/conversations/ or just summarize. The same decision two days earlier chose "pinned"; this time it chose summary and noted "from conversation, summarized (no transcript)." The author subsequently codified this as a rule (template 1.0.4): any capture that names rejected alternatives, records a reversal, or is asked to "pin this" is always pinned. This one named two rejected alternatives, so under the current rule it should be pinned — but the author hasn't re-run it yet.

Same prompt, same commit: 17 pages today, 21 tomorrow. Build is a sample, not a constant.

How to Reproduce

The whole loop, aside from installing the plugin, is three commands plus a loop: after cloning the repo, run the "no wiki" questions in headless mode, initialize and build the wiki, then ask via /llm-wiki:query. Cost is reported by the CLI, and timing is measured by wrapping the call in date.

Key tip: run every question in a fresh session (don't use --continue), otherwise the second question inherits the reading results from the first, making the later-running side look better. If you plan to cite page counts, run the build twice.

The template and plugin used are MIT-licensed, hosted at http://github.com/vellumlabs/llm-wiki-kit.

Headless mode and session isolation: claude -p is Claude Code's headless invocation — it doesn't enter an interactive terminal but takes a prompt and returns a result, suited to scripting, batching, and measurable cost/timing. The article emphasizes running every question in a fresh session and not using --continue, because session continuation preserves previously read file contents and reasoning context — effectively "pre-loading" knowledge into the later-running side, artificially lowering its time and cost and contaminating the controlled experiment. In addition, cost is taken from the CLI-reported total_cost_usd, and timing is measured by wrapping the call in date. The reproducibility of this measurement method depends on fixing three things: model version, prompt, and repo commit. The author notes that two builds of the same commit yield different page counts, so if you want to use page count as a metric, run it repeatedly and rely on checker tools rather than drawing conclusions from a single result.

One-line Summary

Tests show an LLM wiki is slower and pricier than reading source on small codebases, but its value is recording decisions nobody writes down.

Paragraph Summary

Vellum Labs compared an LLM wiki against agentic retrieval on the 1,176-line itsdangerous codebase. On four questions answerable from source, the wiki was 55% slower and 34% more expensive per question in the first round, and the $4.23 build cost can't be recouped by retrieval savings; queries are costly because citations must still be verified against source. But on a code-review decision not in source, docs, or changelog, the wiki became the only option that could answer: capture took 34 s and $0.57 to persist the decision, rejected alternatives, and cost reasoning. The author's boundary follows: use RAG or agentic retrieval when answers live in stable source; use a wiki when knowledge is scattered across people's heads, reviews, and conversations and needs an audit chain and cross-session accumulation; use both for large corpora.