When an AI Agent Trusts a Fake Webpage: $6,000 That Can't Be Recovered, and How to Make It "Forget"

Title

When an AI Agent Trusts a Fake Webpage: $6,000 That Can't Be Recovered, and How to Make It "Forget"?

Meta Description

An AI agent paid $6,000 based on contaminated data, and it can't be recovered. The RETCON project uses the FalkorDB graph database to record the "reasons" behind memories, enabling traceable, recoverable retraction, and quantifies payment fragility to prevent agents from blindly trusting a single source.

Article Content

When an AI agent makes a decision based on contaminated information, simply deleting the source data is nearly useless—all the downstream notes, plans, and actions still exist. This is the "memory retraction" problem that anyone building trustworthy AI agents must face. An open-source project called RETCON offers a solution based on the graph database FalkorDB: let the agent remember the "why," not just the "what."

The Problem: Delete the Source, the Contamination Remains

The story begins with a real lesson. An accounts-payable AI agent, based on information on a supplier portal, paid $6,000 to a tampered bank account. When it was discovered that the portal had been compromised days earlier, the simple fix—deleting that page from the agent's memory—did almost nothing. The page was just where the rumor first appeared. By payment day, another agent had paraphrased it into a note, a third had marked it "confirmed," and the payment agent had combined it with a real invoice, formed a plan, and executed the payment.

Delete the page, and all these downstream memories remain in place, looking completely reasonable. It's like debunking a rumor by burning the newspaper that first printed it.

The core problem is that most memory layers cannot answer: What did my agent believe, plan, and do because of this source? Vector databases can store text and vectors, but they don't record why that content exists.

The Solution: Using a Graph Database to Record the "Reasons" Behind Memories

The RETCON project takes the opposite approach. Every memory an agent writes retains its justification as edges in FalkorDB: which source said it, which agents repeated it, which plans depended on it. Agents never declare these links; the middleware automatically records what was in context at write time. One core rule does most of the work: a memory is believed as long as at least one of its reasons is intact.

This means a fact can have multiple reasons. For example, a supplier's remittance address appears both on the compromised portal and in a signed contract. When the portal is retracted, the address survives because of the contract reason; but the new bank account has only the portal as its source, so it is removed, along with the downstream notes, confirmations, plans, and payment. This process doesn't delete data—it marks it as "OUT," and it can be restored if you change your mind.

In the graph, retraction happens in three steps: mark the source observation as OUT; use a single variable-length match to temporarily mark all downstream memories as OUT; then loop the query until no results are returned. A memory is restored to "IN" only when all the premises of one of its reasons are alive. This prevents two memories that only reference each other from resurrecting each other in a cycle.

The idea isn't new—truth maintenance systems implemented similar logic within a single program back in the late 1970s. What the graph database changes is that "what depends on this source?" becomes a query across all agents and sessions, and the answer includes actions that have already left the system.

FalkorDB is a graph database built on Redis that supports the property graph model and the OpenCypher query language, with built-in graph algorithm modules (such as PageRank, connected components, and maximum flow). Its distinguishing feature is storing graph data as sparse matrices, so queries can directly leverage Redis's in-memory speed. RETCON chose FalkorDB because it satisfies three requirements at once: millisecond-level multi-hop traversal (for cascading retraction), built-in FLOW projections (for maximum-flow computation in fragility analysis), and the GRAPH.COPY command (for safely previewing retraction effects on a replica).

Operations: Preview, Commit, and Restore

The system provides three buttons: Preview runs the retraction on a copy of the graph, shows which memories would be affected, then deletes the copy; Commit performs the actual retraction; Reinstate restores everything exactly.

In the demo, the preview shows: one source, two observations, four beliefs, two plans, three actions. A pending $34,500 payment is blocked. The $6,000 already paid enters a recall list, along with a drafted bank recall request, but nothing is sent until a human approves it. This is exactly what the title means: retraction can correct beliefs in memory, but it cannot recall a wire transfer that has already gone out.

Proactive Defense: Quantifying the "Fragility" of Payments

Once reasons are stored in the graph, you can ask before an attack happens: how many independent sources would an attacker need to compromise to forge this payment? This is a minimum cut problem. The system computes it in two ways: an exact answer derived by evaluating the true belief rules after removing candidate source sets (cheap, because payments rarely depend on many sources); and FalkorDB's built-in maximum-flow procedure, which quickly gives a second number on a dedicated FLOW projection.

A caveat to be honest about: maximum flow treats "needs A and B" as "A or B," so it is exact for simple chains but otherwise only an upper bound. On the smallest demo graph, it gave 3 for a payment whose true cost was 1. So the system reports both numbers, labels the maximum-flow result as a bound, and uses the exact answer for payment gating.

In the agent demo, a $34,500 payment scores 1: one webpage away from fraud. The same is true for other decisions the agent made that day. So the system sets a gate: any payment over $10,000 that relies on fewer than two independent sources is refused.

Minimum cut is a classic problem in graph theory: in a weighted graph, find a set of edges whose removal disconnects the specified source and sink, with the smallest possible total weight. In RETCON's context, each source is treated as an edge, and the weight reflects how hard or independent it is for an attacker to compromise that source; the size of the minimum cut is the lower bound on the number of independent sources an attacker would need to compromise simultaneously. The exact minimum cut is usually solved via the max-flow min-cut theorem, but RETCON points out that because belief rules contain "and/or" logic, maximum flow can only give an upper bound (since maximum flow approximates "needs A and B" as "A or B"), so the system uses exact rule evaluation as the basis for final gating.

Benchmarks: The Limits of Simple Rules

The research team wrote five retraction rules and ran the same attack against all of them: delete the source; delete similar content; invalidate one hop; contaminate all downstream; and their own rule.

The first three rules are easily defeated. In simple scenarios, they leave 87% to 94% of the contamination behind and cannot recall payments already made.

The "contaminate all downstream" rule ties with theirs: zero contamination residue, zero collateral damage, all payments recalled. This made them briefly suspect their project was just a long detour around recursive deletion.

What makes the two rules diverge? Honest data downstream of a bad source that also has a second, independent reason. Simple scenarios almost never contain this, so they built a dedicated scenario: five compromised suppliers, each with its own contamination window, where about 40% of what they touch is also confirmed by a second source.

Rule	Contamination residue (simple scenario, 1024 memories)	Honest payments incorrectly flagged (exact scenario)	Contamination residue (40 real retractions)	Lost topic-level claims across multiple papers
Delete source/similar/one-hop	87-94%	0	7-84%	0/24
Contaminate all downstream	0%	36 ($1,163,150; 18 already paid)	0%	24/24
RETCON	0%	0	0%	0/24

"Contaminate all downstream" flagged 36 correct payments, each of which is an account someone has to reconcile on Monday.

To be fair, the exact scenario is constructed: it tests whether the capability matters, not how often it occurs. In simple scenarios the two rules are indeed identical, and the README says so.

Real data is more interesting. They replayed 40 real retractions with actual dates from the Retraction Watch database. Contaminate-all-downstream lost only 1% of supported memories, because most claims rest on a single paper and should disappear. But that 1% was concentrated in one place: 24 topic-level stances supported by multiple independent papers—exactly the kind of claim a research agent would rely on.

Truth maintenance system (TMS) is a knowledge representation mechanism proposed in the AI field in the late 1970s for automatically updating belief networks when facts or rules change. Its core idea is to record a "justification" for each belief; when a justification becomes invalid, the system recursively retracts beliefs that depend on it, and restores them when an alternative justification appears. RETCON's rule that "a belief holds as long as at least one reason is intact" is exactly the classic TMS logic—the difference being that RETCON extends the justification graph to a distributed memory layer across agents and sessions, whereas traditional TMS runs within a single program's internal data structures.

Self-Doubt: The Wariness That Comes with a 0% Score

RETCON scored zero on both contamination and collateral damage, which made them suspicious of themselves. The benchmark's "ground truth" and the cascade logic share the same definition of "contamination"—essentially writing the answer themselves.

So they trusted a different number more: they wrote a deliberately naive second implementation (about 60 lines, no cascade) and made the real engine agree with it on 1000 random graphs and hundreds of random retraction/restore sequences. This checker found an engine bug almost immediately, and following it led to a second: restoring one retraction would resurrect observations still covered by another overlapping retraction; and restore failed to bring back a memory that had been flipped by a different retraction but had regained a reason in the meantime. Both bugs escaped unit tests. The checker convinced them that the zero described the engine, not the test data.

Comparison Without the Graph, and Model Boundaries

They ran the same agent and the same attack in a setting where similarity search replaced reasons. Result: 3 of 9 bad memories retracted, 5 good memories lost. With the graph: 10 of 10 retracted, zero lost. (9 versus 10 is not a typo: without the graph there's no gate, so the agent behaved slightly differently that day, and each run is scored against its own graph.)

They had wanted to title it "Graphless Agents Pay the Attacker." That was true in the offline demo (using a simple hash embedder), but not with a real embedding model. So the honest conclusion is narrower: memory without reasons forgets too little, and forgets the wrong things. It doesn't always pay. They changed the title.

The division of responsibility is clearly drawn: the payment agent uses Gemini function calling to choose tools (check fragility, read the reason tree, hold or pay), but never decides what to believe. That's the graph's job. If the model tries to pay against the graph, the gate blocks it. The agent isn't perfectly stable: under a looser prompt, it once withheld a payment that should have been made, and once wrote "request callback verification" without setting a send flag. The team tightened the prompts, committed all 49 calls, and the demo replays the recording of that correct run—no API key needed.

Other Lessons and Open Questions

A few more issues came up along the way: the initial version of scientific claim extraction found that 18% of a random sample of 50 were not claims; the abstract data source once returned completely unrelated abstracts; they hit a known FalkorDB edge case (single-node projections caused algo.pageRank and algo.WCC to error, already reported as issue #3349) and worked around it by counting nodes and skipping the call; a regex replacement turned \1 \2 into two control characters, silently corrupting the text; and a payment with no amount sorted before $6,000 in the recall list (the ordering behavior of null under DESC).

RETCON is slower than naive rules: on a graph of nearly ten thousand memories, one retraction takes 47 milliseconds versus a few milliseconds for the naive rules. With 50 agents writing concurrently, 1620 operations left no stale answers. Fast enough, but not faster—the extra time buys the second-source check.

Advice to their day-one selves: build the naive reference checker first—it found more bugs than all the unit tests combined; design a benchmark you can lose; keep the model away from the truth—let the LLM choose actions and let the graph decide what to believe.

One claim they don't make: "contaminated" means "rests only on a source you no longer trust," not "false." A background sentence from a retracted paper may be completely true.

Trying It Out

Clone the repository and run docker compose up --build, then open 127.0.0.1:8200 to run the steps above against the real engine. Every number on the homepage comes from result files. It's also a regular MCP server providing 15 tools; Claude Code can ask which source should be audited first and preview a retraction.

Agents are getting better and better at learning. The harder part is giving them a way to stop believing, and to tell you what that belief has already done.

One-line Summary

RETCON uses a graph database to record the "reasons" behind AI agent memories, enabling precise retraction and proactive defense.

Paragraph Summary

RETCON is an open-source project built on the FalkorDB graph database, designed to solve the "memory retraction" problem where an AI agent makes a wrong decision based on contaminated information, and simply deleting the source cannot eliminate downstream effects. It records the source and derivation relationships for each memory (as edges) and follows the rule that "a belief holds as long as at least one reason is intact," enabling cascading retraction and restoration. The system provides preview, commit, and restore operations, and can compute payment fragility (minimum cut/maximum flow) to set gates. Benchmarks show that RETCON outperforms simple rules at both clearing contamination and protecting honest data, especially avoiding the $1.16 million in honest payments wrongly flagged by "contaminate all downstream." The core design principle is to let the LLM choose actions and let the graph decide what to believe.