Provenance Gate: No AI Agent Memory Write Without a Source

Most AI agent memory systems trust anything the agent writes by default—and that's exactly the problem. The agent infers a number from context, writes it into memory with full confidence, and a month later you can no longer tell whether that information was something the customer actually said or something the model made up. Mikhail Oskola, founder of Retailbox Automation, implemented a "provenance gate" in the hackathon project FleetMemory, then attacked his own system personally and found that the first version was breached by all 6 attacks. After the fix, all 30 attacks were blocked.

What Is a Provenance Gate

A provenance gate is a rule standing in front of AI agent memory: the agent may propose a fact, but unless it specifies where that fact came from, it cannot be written to memory. No source, no write.

This check does not compare whether the value is true or false; it only looks at whether a source is cited. When a customer says "our budget is now $14,000," that has provenance and can overwrite an old value. When an agent infers a budget number from conversational context while no one actually said that sentence, there is no source, so the claim is quarantined rather than written directly.

In the FleetMemory demo, a rogue agent asserts a budget of $1,800 through exactly the same path as normal writes. The gate cannot determine that the number is false; it merely checks whether a source is cited. So the claim is quarantined under a golden "pending review" seal, and then an adversarial verifier reads the case and rejects it, while directly above it sits the real fact that passed the gate: $18,000, sourced from a real customer message.

The gate actually enforces four classes of checks: junk and duplicate checks, where malformed or known writes are discarded before consuming verifier calls; contradiction checks, where new values that conflict with existing facts are flagged rather than silently overwritten; a confidence floor, where guesses below the minimum confidence never enter memory; and provenance checks, where claims unsupported by any customer utterance are quarantined no matter how confident the agent sounds.

Confidence in LLM applications usually refers to a model's probabilistic assessment of its own output, obtainable through logprobs, self-evaluation prompts, or dedicated calibration models. However, high confidence does not equal high truthfulness—a model can state a fabricated number with extremely high confidence, especially when such numeric patterns are common in training data and the current context lacks comparable anchors. This is precisely the root cause of the first gate version's failure in FleetMemory's red-team test: when there is no existing fact in memory to contradict it, a pure confidence filter has no comparison baseline and can only accept passively. Therefore, provenance checks and the confidence floor are not substitutes but complementary lines of defense: confidence filters out obviously unstable, low-quality guesses, while provenance intercepts assertions that are confident but unsupported.

Vulnerabilities Found by Red Teaming

The quality of a gate depends on what you throw at it. The team ran 30 adversarial writes covering 5 attack types against the real gate and verifier (not mocks). The first run exposed a real gap: novel claims without provenance—facts that did not contradict any existing fact in memory because there was simply nothing to contradict—passed the deterministic gate at a rate of 6/6. A brand-new, confident, completely fabricated fact walked straight into shared memory.

The fix was a rule the team called needs_provenance: claims unsupported by customer utterances are quarantined and subjected to verifier review, and claimed confidence alone can no longer automatically overwrite existing sourced facts. Rerunning the same 30 attacks, all were blocked this time.

The gate still let through 9 of 10 legitimate customer-confirmed updates, rather than all of them. The one it missed was a disclosed race condition: a value cycled back to its original value while an intermediate update was still unresolved. This case was recorded in the decision log, not hidden or papered over with tuning. Median gate latency across the whole run was 293 milliseconds.

Adversarial Write refers to deliberately constructed memory write requests in red-team testing that simulate malicious or rogue agent behavior, with the goal of finding exploitable paths in the real gate and verifier rather than deceiving oneself in a mock environment. The five attack types covered by the FleetMemory team typically include: novel claims without provenance, overwrite attempts without provenance, low-confidence high-risk values, format-disguised injection, and duplicate poisoning. The first gate version let through 6/6 of the "novel claims without provenance" type, showing a structural blind spot in deterministic checks—they can only compare against existing facts and cannot handle facts appearing for the first time with no external citation. The value of this testing method is that it forces the system to expose real gaps before deployment, rather than reverse-engineering them from production incidents after the fact.

Why the Verifier Denies by Default

Quarantined claims do not just sit there. An adversarial LLM verifier reads them and makes a ruling, and its default ruling is to deny rather than accept. If the verifier call itself fails for any reason, that also counts as a denial.

The verifier's responsibility is specifically to weigh provenance: direct customer statements can overwrite old facts; values without provenance are rejected—even if they sound reasonable, even if the agent stating them sounds confident. Every ruling, along with a one-sentence rationale, is written back to the same decision log line as the original write attempt.

Corrective operations also do not delete anything. An overwritten fact retains its old row, marked invalid and pointing to the fact that replaced it. Ask the system what it believed on any historical date, and it can tell you, including beliefs it later abandoned.

Default-Deny is a security design principle derived from whitelist strategies in firewalls and access control: unless the conditions for allowance are explicitly met, deny. In the context of FleetMemory's verifier, this means that if a quarantined claim cannot prove that its source is a real customer utterance, the verifier rejects it, even if its value sounds reasonable and even if the agent proposing it appears very confident. Equally important, if the verifier call itself fails—timeout, network error, abnormal model return—that also counts as a denial, rather than being silently allowed through or retried and then accepted. This design systematically pushes "uncertainty" toward the conservative side and avoids silent contamination during failures. The cost is that it may wrongly reject a small number of legitimate updates, such as the race condition case, but the team chose to record and disclose this cost rather than hide it through tuning.

The Strongest Proof: Someone Else Independently Verified It

Anyone can claim their system works. During development, the team found that LangChain's AsyncPostgresSaver checkpoints failed on CockroachDB, and submitted an upstream issue with an independent reproduction and a direction for a fix (langchain-ai/langgraph#8620).

A community contributor picked up the problem, built a complete fix along that direction, and independently verified it on PostgreSQL 17.5 and CockroachDB v26.2.4. It is currently awaiting maintainer review. No one on the team wrote that fix or ran that verification—which is more persuasive than any benchmark they could publish themselves.

LangGraph's AsyncPostgresSaver is a checkpoint storage component in the LangChain ecosystem used to persist agent state to PostgreSQL, supporting asynchronous operations, but it has compatibility defects on CockroachDB. Although CockroachDB is compatible with the PostgreSQL wire protocol, its distributed transaction implementation differs subtly from PostgreSQL in isolation levels, serialization conflict handling, and system table behavior. These differences are enough to make checkpoint logic designed for single-node PostgreSQL fail in a distributed environment. The upstream issue submitted by the team (langchain-ai/langgraph#8620) not only includes reproduction steps but also gives a direction for the fix, which is a typical model of efficient collaboration in the open-source community: the person who discovers the problem provides sufficiently precise clues, and an independent contributor follows those clues to complete the fix and verify it in both PostgreSQL 17.5 and CockroachDB v26.2.4. This kind of third-party verification is more persuasive than self-run benchmarks because neither the fixer nor the verifier has an incentive to beautify the results, and the verification environment is independent of the original project.

Where to See It Running

FleetMemory is the team's entry for the CockroachDB x AWS "Build with Agentic Memory" hackathon, not a paid client project. It is public and MIT-licensed, so anyone can read the gate logic or rerun the red-team tests.

Architecturally, every write by agents Alex and Maria (built on LangGraph) passes through the write gate; quarantined writes go to a default-deny verifier sub-agent; facts, the C-SPANN vector index, and the gate_decisions log are stored in CockroachDB; and AgentCore Memory, Bedrock, and Lambda demos run on AWS.

There is a live demo with no login required. You can talk to Agent Alex, come back as Agent Maria, and then click "hallucinate a fact" to see how the gate catches it. The public repository contains the gate, verifier, red-team scripts, and evaluation results. There is also a 1-minute-15-second demo video walking through the moment of quarantine and rejection.

How This Differs from RAG Citations

The relationship is that these are earlier links in the same direction. RAG citations tell the user where the text of an answer came from; a provenance gate decides whether a claim is eligible to become a durable fact in memory—before any agent retrieves and repeats it.

Confidence scores are also insufficient to filter out bad facts. This is exactly the gap red teaming found: an agent can state a brand-new, unsourced fact with high confidence, and if there is no existing content in memory to contradict it, a confidence-only filter has nothing to compare against, so the fact slipped past the first gate version at 6/6. The fix checks provenance, not the confidence number.

Facts rejected by the gate do not disappear. They are written to the decision log with the reason for rejection, so anyone can query what was attempted, when, and why it failed.

The red-team scripts are MIT-licensed, and you can point them at your own CockroachDB cluster to rerun the same 30 attacks.

One-line Summary

AI agent memory provenance gate: unsourced claims are quarantined, and all 30 red-team attacks are blocked.

Paragraph Summary

Most AI agent memory systems trust anything the agent writes by default, making it impossible to distinguish customer statements from model fabrications. Mikhail Oskola, founder of Retailbox Automation, implemented a "provenance gate" in the hackathon project FleetMemory—agents may propose facts, but must specify a source, or the write is quarantined. The gate enforces four checks: junk/duplicates, contradictions, a confidence floor, and provenance. In the first red-team run, novel unsourced claims passed 6/6; after the needs_provenance fix, all 30 attacks were blocked while still allowing 9/10 legitimate customer updates. Quarantined claims are adjudicated by a default-deny adversarial verifier, with results and reasons written to the decision log; overwritten facts are retained and marked invalid, allowing the system to reconstruct beliefs on any historical date. Unlike RAG citations, the provenance gate acts earlier in the pipeline, deciding whether a claim is eligible to become a durable fact. The project also found and fixed a LangChain checkpoint issue on CockroachDB, independently verified by a community contributor.