AI Memory Correction Cites the Old Error: How a Graph Database Fixes Provenance

Correcting a memory error for an AI agent, you think you're done — until it takes your freshly corrected conclusion and cites that original error as evidence, confidently telling you at confidence 1.0: "This time it's definitely right." This is the real dilemma developer Vaibhav encountered while building the Afterhack project at the WeMakeDevs Graph Hacks hackathon, and it's a side that almost every agent-memory demo never shows.

Why Vector Retrieval Can't Save "Correction"

The common agent-memory approach is chunking, embedding, and similarity retrieval. That works fine for finding information, but it can't answer a key question: "If this fact is wrong, what else is wrong along with it?"

The reason is that similarity is not dependency. Two text chunks may both talk about the same product launch, but one isn't necessarily the reason the other holds. What Vaibhav needed was a clear "because" chain:

Source ──▶ Fragment ──▶ Claim ──▶ Decision ──▶ Action
                          ▲
                      Correction
         (everything to the right of it might now be wrong)

Once the problem is written this way, "what did this wrong fact affect" shifts from a search problem to a graph traversal problem. That's exactly why he chose FalkorDB — each workspace uses its own separate graph, with the graph name derived from a server-generated UUID. A request cannot specify which graph to access, which prevents unauthorized access at the isolation level.

The core difference between graph databases and vector databases lies in the data model and query paradigm. A vector database encodes text into high-dimensional vectors and returns semantically similar entries via approximate nearest neighbor search (ANN). It excels at "finding similar things" but cannot express directed relationships between entries. A graph database stores entities and their associations as nodes and edges, supporting multi-hop traversal, path queries, and pattern matching (such as Cypher's MATCH, -[:REL]->), and can precisely answer provenance questions like "which nodes depend on this node." In this scenario, "claim → decision → action" forms a directed dependency chain, and a correction must propagate its impact along the chain — exactly where graph traversal shines. Vector retrieval, by contrast, can only return "text similar to the erroneous claim" and cannot distinguish similarity from dependency, so it cannot perform invalidation propagation.

Rules First: No Exact Citation, No Evidence

Before considering correction, the first priority is ensuring every claim stored in memory is trustworthy. To that end, the extraction stage sets a hard rule: the model must return an evidence_quote for each claim, and the API validates that the quote is an exact substring of the source document — character-for-character.

If the model paraphrases the original, discard it. If it fabricates a quote, discard it. Claims that pass validation are stored together with start and end offsets, so the UI can precisely highlight which characters of the original text the sentence came from.

This rule is blunt and will wrongly reject some technically valid claims. But the author is at peace with that: an occasionally forgetful memory system is annoying, but a confidently fabricating memory system is far more terrifying.

Exact substring matching is a strict form of evidence validation: the quote text output by the model must exactly match some stretch of characters in the source document, including spaces, punctuation, and case. Unlike fuzzy matching or semantic similarity, exact matching cannot be satisfied by paraphrases that are "similar in meaning but different in wording," so it effectively blocks the model from restating or polishing the original from memory. Implementation typically uses string search (such as Python's in or str.find) and records the match's starting offset to support UI highlighting. The cost is lower recall — some semantically correct claims whose wording was adjusted will be wrongly judged invalid. This "rather kill than let slip" design puts evidence reliability above coverage, which suits memory systems with high requirements for factual accuracy.

A Search Hit Is Not Evidence: Blocking the Model's Shortcuts with Code

The agent side is divided into two roles, composed of two model calls. The Analyst has the ability to search claims and pull original fragments, and hands its proposal to the Coordinator, which reviews it and writes a decision draft.

The first real test exposed two problems. First, the Analyst searched entire sentences the way a person uses Google (e.g., "launch date for the product"), while the Cypher query used CONTAINS, so it naturally found nothing, and the agent concluded "there is no evidence." A fallback strategy was added: if the full sentence misses, split it into content words and remove stop words, then search again.

The second was more serious: the Analyst sometimes found the claim but skipped the step of pulling the source fragment, drawing conclusions directly from claim IDs. The author therefore wrote the project's motto: "A search hit is not itself evidence," and enforced it at the code level:

allowed_ids = searched_claim_ids & retrieved_claim_ids
claim_ids = [str(i) for i in proposal.get("claim_ids", []) if str(i) in allowed_ids]

The Coordinator can only cite claims that were both searched and had their source fragments pulled during this run. It cannot fabricate IDs, nor reuse old IDs from memory. If none survive the filter, the decision is marked needs_review with the title "No source-backed answer yet," and no action is produced.

The author tested with a deliberately vague prompt in a workspace with no relevant evidence; the agent honestly answered "I don't know" and stored that trajectory along with the handoff record in the graph database. That, ironically, became his most satisfying result of the entire weekend.

The CONTAINS operator in Cypher performs a case-sensitive substring containment test, not full-text search or semantic search. When the Analyst passes an entire natural-language sentence (such as "launch date for the product") directly as a CONTAINS argument, it demands that the text of some claim in the graph fully contain that whole sentence, which is nearly impossible to hit in real data. The essence of the fallback strategy is query rewriting: split the sentence into content words, remove stop words (such as "for" and "the"), and then match each one using multiple CONTAINS conditions or tokenized keywords. This exposes the semantic gap between LLM-generated queries and database query languages — the model constructs input according to human search habits, while the database executes according to exact string rules; when the two don't match, an explicit adaptation layer is needed.

The "Oh..." Moment: The Correction Inherits the Wrong Evidence

The correction flow was supposed to work like this: a reviewer enters the corrected statement, Preview shows the decisions and actions that depend on the claim, and Apply marks the old claim as superseded, creates a new claim with a SUPERSEDES relationship, and updates the affected decisions.

The feature was built, and the replacement statement text was correct. But when the author checked what supports this new claim, the problem appeared — the first version of the code copied the old claim's evidence directly onto the new claim:

CREATE (replacement:Claim {
  statement: $statement,
  evidence_quote: old.evidence_quote,  -- 👀
  confidence: 1.0,
  ...
})
CREATE (replacement)-[:SUPPORTED_BY {role:'correction_context'}]->(f)  -- the OLD fragment

So the replacement claim said "July" but pointed to a document that said "June" as proof, with confidence maxed out. When the agent later retrieved it, it found this "corrected" claim, pulled its "source," and then quoted the original error back as the basis for the fix. Every step faithfully executed instructions; the graph database simply faithfully preserved a lying link.

The fix was to treat the correction itself as a new source of evidence:

CREATE (f:SourceFragment {
  text: $statement,
  origin_type: 'human_correction',
  origin_name: 'Workspace reviewer correction'
})
CREATE (corr)-[:HAS_FRAGMENT]->(f)
CREATE (replacement:Claim {
  statement: $statement,
  evidence_quote: $statement,
  confidence: 0.0,
  confidence_kind: 'human_correction',
  ...
})
CREATE (replacement)-[:SUPPORTED_BY {role:'reviewer_correction'}]->(f)

Corrections now create an independent fragment attached to the Correction node and marked as a human source, and the replacement claim cites this fragment. Confidence starts at 0.0 and carries an explicit confidence_kind — a sentence typed by a reviewer and a model's match against a quote are not the same thing, and the author didn't want a single number pretending to express both.

The SUPERSEDES relationship represents version replacement in the graph model: the new node points via this edge to the superseded old node, which is retained but marked superseded, thereby maintaining a complete history chain rather than physically deleting it. This design supports time-travel queries ("what did the system base its decision on before a certain correction") and audit provenance. Crucially, SUPERSEDES only expresses "who replaced whom" and does not automatically transfer evidence relationships — the new claim must point to its own evidence fragment through an independent SUPPORTED_BY edge. If an implementation mistakenly copies the old claim's evidence edges onto the new claim, it creates the erroneous combination of "new statement + old evidence," and the graph database won't proactively detect this semantic contradiction because it is structurally perfectly valid.

Related Does Not Mean Invalid: Don't Let Alarms Cry Wolf

After fixing the evidence problem, a second error surfaced. The original propagation rule was crude: any decision that used a corrected claim was marked needs_review, and actions depending on that decision were marked held.

This leads to an explosion of false positives. Imagine a decision supported by three claims, one of which is corrected while the other two still fully support it. Flagging it is like a smoke alarm going off nonstop while toasting bread — after the third time, no one looks anymore.

The corrected logic is: only mark a decision for review when it loses its last supporting claim, and only suspend actions that depend on those decisions.

When the Test Database Is Fake, You're Testing Your Assumptions

The first version of the propagation logic was an elegant query using a NOT EXISTS { ... } subquery to find, in one go, decisions with "no other supporting claims." The unit tests passed — because they used fake objects and never ran the real Cypher.

The real correction test returned HTTP 500 outright. FalkorDB doesn't support that subquery form. The author had to split it into multiple bounded, read-only queries that FalkorDB definitely supports, and only then did the full end-to-end flow pass. His conclusion was blunt: test doubles test your assumptions about the database; only tests against the real database catch bugs like this.

Trade-offs Between Concurrency and Atomicity

Corrections change shared state, so concurrency must be handled. The workspace and each claim carry a revision number. Preview returns the version it saw, and Apply must send it back; the write only takes effect if the version still matches. If something changed in the meantime, it returns 409 stale_correction_preview; if the same correction was already applied, it returns the existing replacement claim marked already_applied and reruns the dependency check to guard against a crash midway through the first attempt.

The author also admits a trade-off: correction plus repair involves multiple FalkorDB calls and is not a single atomic transaction. If the process crashes between "mark the claim superseded" and "suspend the action," partial state may remain. Retries can clean it up, but that is a recovery path, not a guarantee.

Hide the Database from Users

The last discovery has nothing to do with AI. When the full graph was initially rendered in the UI, Workspace, SourceVersion, SourceFragment, AuditEvent, AgentRun, and Handoff were everywhere, like a crime-scene evidence board — technically accurate, and utterly useless to someone asking "what happens if this fact is wrong."

The backend was therefore changed to return a product-level view: Sources → Claims → Decisions → Actions, plus Corrections, with relationship names swapped for readable words like SUPPORTS. The underlying structure still lives in FalkorDB for provenance, but users don't have to read those pipes. The author uses this to make a point: the hard part of graph visualization is deciding what not to show.

Current Limitations and Core Lessons

The project still has several unfinished areas: in real tests the Coordinator never actually proposed an action, and "suspend action" was only verified in storage-layer tests; "alternative support" only means at least one other current claim, without asking whether it comes from an independent source or tracking multi-level chains; each source has only a single fragment, not yet chunked; and the repair process is not atomic.

The author offers four lessons to his Friday-night self: a correction is evidence, not an edit — if you let the fix inherit the wrong provenance, the error never really leaves, it just gets a new title; enforce evidence in code, not by requesting it in the prompt — "please only cite retrieved evidence" is a request, searched & retrieved is a rule; related does not mean invalid — flag everything and people learn to ignore flags; print things out — the header bug, the NOT EXISTS 500 error, the copied quote: each time he had assumed what something looked like instead of actually looking.

What this project ultimately wants to say is: the interesting part of agent memory isn't remembering, but what happens after it remembers wrong, and how much has already acted on that error.