Your AI Knowledge Base Had 61 Pages. 24 Were Obsolete: The Hard Truth About RAG Document Rot

A dental clinic's AI receptionist kept quoting a "wrong" price. It wasn't wrong, exactly — it was the price from two years ago, hiding in a help-center article nobody had deleted.

I audited the clinic's entire knowledge base: 61 Markdown pages, last modified anywhere from 14 to 1,100 days ago. I compared every page against the live systems — the appointment calendar, the price list, the staff roster, the insurance list.

24 of the 61 pages (39%) contained at least one statement that was already false.

Six mentioned a dentist who no longer worked there; five quoted prices that had changed; four described services the clinic no longer offered; three had opening hours that were last correct before 2024; two listed insurers they no longer worked with; four were duplicates of other pages that gave a different answer to the same question.

The AI retrieved these pages with a confidence of 0.94, because retrieval measures similarity, not truth. A page saying "checkup £45" is a perfect match for the query "how much is a checkup?" It doesn't know the price is now £62.

This is document rot, and in RAG systems it is the single largest source of confident wrong answers. No amount of prompt engineering fixes it.

Four Types of Rot, and Only One Is "Out of Date"

Most teams treat this as one problem — "we should update the docs." It isn't. Each type needs a different fix.

Rot type	Symptom	Why retrieval can't catch it	Fix
1. Stale fact	Price, hours, name, or policy changed; page didn't	Old text matches the question semantically	Timestamp + owner + probe
2. Ghost page	Service, product, or person no longer exists	Still ranks first for related questions	Delete, don't archive
3. Fork	Two pages give different answers to the same question	Retrieval returns the higher-ranked one; both look authoritative	Merge into one canonical page
4. Orphan	Product or site isn't linked and has no owner	Still indexed, still retrievable	Retire or assign an owner

The clinic's 24 bad pages broke down as: 11 stale facts, 6 ghosts, 4 forks, 3 orphans. Only the first category can be solved by "reminding people to update the docs."

Ghost pages are the most dangerous. A page for a discontinued service doesn't just cause wrong answers — it makes customers ask for things they shouldn't be able to ask for. The AI offers a treatment the clinic can't perform, and the customer books it.

Why "Quarterly Doc Reviews" Never Work

I've run this audit on seven knowledge bases, ranging from 61 to 840 pages, mostly 100–300. The results are consistent.

Metric	Typical range across 7 audits
Pages with at least one false statement	31% – 44%
Pages not edited in 12 months	55% – 78%
Duplicate-topic forks	6% – 19% of all pages
Median age of the 20 most-retrieved pages	210 – 640 days
Pages with an empty "owner" field or a departed employee	40% – 66%

Note the last row, and the fourth. The most-retrieved pages are usually the oldest, because they were written when the product launched and nobody rewrote them. Across almost every knowledge base I've opened, retrieval volume is inversely correlated with freshness.

Quarterly manual reviews fail for a structural reason: you can't review 200 pages by reading them, because "is this still true?" cannot be answered from the page itself. It can only be answered from the system of record — the price list, the calendar, the CRM. A reviewer reading a doc can only judge whether it sounds current.

So the audit has to be built as a diff against systems, not a reading exercise.

RAG and the similarity mechanics of vector retrieval

Before answering a user question, a RAG (Retrieval-Augmented Generation) system splits knowledge-base documents into chunks and converts them into vector embeddings stored in a vector database. When a user asks a question, the system converts the question into a vector too, finds the closest document chunks using measures like cosine similarity, and passes them to the LLM to generate an answer. The key point: similarity measures "semantic closeness," not "factual correctness." A page with an outdated price will earn extremely high retrieval confidence as long as its wording closely matches the user's question — the clinic case in this article hit a confidence of 0.94 while its content was a two-year-old price. That means the model will faithfully generate fluent answers from wrong material, and no prompt engineering can make it doubt a document that reads perfectly. The rot originates in the retrieval layer, not the generation layer.

Three Things Every Page Needs Before It Can Be Audited

You can't detect rot without a baseline. Add three fields to every page's front matter, including the pages you're about to delete.

---
owner: revenue-ops            # a team, not a person who might quit
source_of_truth: pricing-sheet-v3   # where the facts come from
verified_at: 2026-09-30       # last time a human or script confirmed it
---

• owner must be a team or role. Every knowledge base I've audited that used individual names as owners had over 40% of its pages become orphans within a year, because when people leave, nobody inherits their docs.
• source_of_truth makes automated checks possible. If a page's price claims can't be traced to a named table, they can't be verified — only believed.
• verified_at is not updated_at. Fixing a typo is not verification. That distinction is the heart of the whole audit.

Pages without these three fields are, by definition, unowned and unverifiable. Treat them as suspects, not documents.

Front matter and the system of record

Front matter is the YAML metadata block at the top of a Markdown file enclosed in ---, usually holding structured information like title, date, and tags, and not rendered as body text. The article's requirement to add owner, source_of_truth, and verified_at to front matter essentially turns documents into auditable objects. The "system of record" refers to the authoritative sources of truth in an organization — price lists, appointment calendars, CRMs, staff rosters. Statements in documents are only snapshots of the system of record, and snapshots drift. Only by diffing page claims against the system of record can you answer "is this still true?" The source_of_truth field exists precisely to establish a traceable link between docs and the system of record, making automated verification possible; without it, page contents can only be believed, not checked.

The Scanner

It runs in about four seconds on 300 pages and produces a worklist. It is deliberately simple: no LLM, no embeddings. It finds candidates; a human confirms.

# rot_audit.py — python3 rot_audit.py ./kb > rot_report.csv
import os, re, sys, csv, datetime, collections

ROOT = sys.argv[1]
STALE_DAYS = 180
NUM = re.compile(r'(?:[$£€]\s?\d[\d,.]*|\b\d+(?:\.\d+)?\s?%|\b\d{1,2}:\d{2}\b)')

def front_matter(text):
    m = re.match(r'^---\n(.*?)\n---\n', text, re.S)
    if not m: return {}
    out = {}
    for line in m.group(1).splitlines():
        if ':' in line:
            k, v = line.split(':', 1)
            out[k.strip()] = v.strip().strip('"\'')
    return out

rows, seen = [], collections.defaultdict(list)
today = datetime.date.today()

for dirpath, _, files in os.walk(ROOT):
    for f in files:
        if not f.endswith(('.md', '.mdx')): continue
        p = os.path.join(dirpath, f)
        text = open(p, encoding='utf-8').read()
        fm = front_matter(text)
        body = re.sub(r'^---\n.*?\n---\n', '', text, flags=re.S)
        nums = NUM.findall(body)
        verified = fm.get('verified_at')
        age = ''
        if verified:
            try: age = (today - datetime.date.fromisoformat(verified)).days
            except ValueError: age = 'BAD_DATE'
        rows.append({
            'file': os.path.relpath(p, ROOT),
            'owner': fm.get('owner', ''),
            'source_of_truth': fm.get('source_of_truth', ''),
            'verified_at': verified or '',
            'days_since_verified': age,
            'numeric_claims': len(nums),
            'stale': 'YES' if (not verified or (isinstance(age, int) and age > STALE_DAYS)) else '',
            'risk_score': 0,
        })
        # fork detection: same title-ish heading, different numeric content
        head = (fm.get('title') or body.strip().splitlines()[0:1] or [''])[0]
        key = re.sub(r'[^a-z0-9 ]', '', str(head).lower())[:40].strip()
        if key: seen[key].append((os.path.relpath(p, ROOT), tuple(sorted(nums))))

for r in rows:
    score = 0
    if r['stale']: score += 2
    if not r['owner']: score += 2
    if not r['source_of_truth']: score += 2
    score += min(len(re.findall(r'(?i)\b(currently|now|as of|starting|from january|this year)\b', open(os.path.join(ROOT, r['file']), encoding='utf-8').read())), 3)
    r['risk_score'] = score

for key, items in seen.items():
    if len(items) > 1 and len({i[1] for i in items}) > 1:
        for f, _ in items:
            for r in rows:
                if r['file'] == f: r['risk_score'] += 3

rows.sort(key=lambda r: -r['risk_score'])
w = csv.DictWriter(sys.stdout, fieldnames=list(rows[0].keys()))
w.writeheader(); w.writerows(rows)

It flags:

• stale — no verified_at, or past the threshold.
• No owner / no source of truth — unverifiable pages score higher and rise to the top so you read them first.
• Temporal language — "currently," "now," "as of" in a page are self-declared expiry dates.
• numeric_claims — every number is a fact that can drift; zero numbers means low blast radius.
• Forks — two pages with the same title but different numbers, a class humans never catch by reading.

The Probe Set: 12 Questions for Both the AI and the System of Record

The scanner finds suspicious pages. Probes find actual lies. Write 12 questions whose answers are findable in your systems — not opinions, not summaries. For the clinic:

1. What does a standard checkup cost?
2. What does a composite filling cost?
3. What are Saturday opening hours?
4. Which dentists are currently accepting new patients?
5. Do you accept [insurance provider]?
6. How long is the wait for a hygiene appointment?
7. What is the cancellation window and fee?
8. Is [discontinued service] available?
9. What is the emergency out-of-hours phone number?
10. How much is a child's first consultation?
11. Which locations are open on Sundays?
12. What is the deposit for treatment plans over £1,000?

Ask the deployed AI each question verbatim. Put its answer side by side with the system-of-record answer. Every mismatch is a rot incident with a fingerprint: the answer the AI gave tells you which page produced it.

At the clinic, the 12 probes returned 7 mismatches, mapping to 9 of the 24 bad pages. The other 15 pages had no probe coverage — which is itself a finding: a page no probe touches is a page you have no reason to keep.

Run the probes weekly. It takes fifteen minutes, and it catches drift in days instead of at the first angry customer.

Probes as end-to-end verification

A probe is a pre-designed question whose answer can be definitively found in the system of record, used to ask both the deployed AI and the authoritative source and compare the two. It differs from the scanner in that the scanner is static analysis, finding candidates that "may be rotten," while probes are dynamic tests that directly expose "actually wrong" outputs. Probe questions must satisfy two conditions: the answer is a fact rather than an opinion or summary, and the answer is uniquely determinable from the system of record. Every mismatch constitutes an attributable rot incident — the wrong answer the AI gives can itself trace back to the page that produced it. The clinic in this article got 7 mismatches from 12 probes, mapping to 9 of the 24 bad pages; the remaining 15 had no probe coverage at all, suggesting they may have no reason to exist. Running the probes weekly takes fifteen minutes and catches drift before customers complain.

Triage: What to Actually Do With Each Finding

The instinct is to fix everything. Don't. Fix in this order, and choose the action from the row's category.

Finding	Action	Time
Ghost page (service/person gone)	Delete the file. Don't archive — archived files get re-indexed.	2 min
Fork (two answers, one question)	Pick the canonical page, paste in the other's unique content, delete the duplicate	15 min
Stale fact with a named source	Correct from the source, update verified_at	5 min
Stale fact with no source	Find the owner, get the number, write it in the source system first	30 min
Orphan with no owner	Assign an owner or delete. There is no third option.	5 min
Out of date but still true	Update verified_at and record that it was checked	1 min

The clinic's 24 pages took 3 hours 40 minutes in total, most of it on the six forks. Two weeks later, re-running the 12 probes returned 0 mismatches, and the "wrong price" complaints stopped.

Re-indexing matters: after any deletion, force a re-embed. A deleted file still in the vector store is a ghost that escaped the exorcism.

Re-embedding and vector store residue

Deleting the source file does not mean deleting its corresponding embedding from the vector database. A RAG system's vector store is stored separately; after a document is deleted, its embedding may remain in the index and continue to be retrieved — what the article calls "a ghost that escaped the exorcism." So after every deletion or page merge, you must force a re-embedding process to keep the vector store in sync with source documents. This is partly why orphan pages are dangerous: even with no one maintaining them, they are still indexed and still retrievable. Likewise, ghost pages must be deleted rather than "archived," because archived files are often still picked up by the indexing pipeline. Re-embedding should be written into your ops process as a standard follow-up to deletion, not an optional step.

The Monthly Numbers That Tell You It's Working

Month: ____________________
Total pages:                    ____
Pages verified this month:      ____   (target: ≥25% of KB, full cycle ≤4 months)
Pages without an owner:         ____   (target: 0)
Active probe questions:         ____   (target: ≥12)
Probe mismatches:               ____   (target: 0)
Median days-since-verified, top 20 retrieved pages: ____ (target: <90)
Ghost pages deleted:            ____
Forks merged:                   ____

The median-days row is the one to watch. If your 20 most-retrieved pages are more than 90 days unverified, your AI is confidently describing an old version of your business — and you'll find out from customers, not from a dashboard.

The Ten-Line Check Before You Trust a Knowledge Base

[ ] Every page has owner, source_of_truth, verified_at
[ ] No owner is a person's name
[ ] Every page with a number says where the number comes from
[ ] No page says "currently," "now," or "as of" without a date
[ ] No two pages give different answers to the same question
[ ] No page describes a service you can't buy today
[ ] No page mentions a person who no longer works there
[ ] 12 probes are defined, each answerable from the system of record
[ ] Probes run weekly against the live agent, mismatches triaged within 48h
[ ] Re-embedding is forced after every deletion

None of this is prompt engineering. Prompt engineering changes how the model talks. Document rot changes what it believes, and no instruction makes a model doubt a document that reads perfectly.

The clinic's fix wasn't a better prompt. It was deleting 24 files and adding three lines of front matter.