AI Agent Memory Audit: 9 of 64 Stored Entries Failed, and the Problem Was the Test Method

Long-horizon AI agents' public traces almost all share the same blind spot: they record conversations, tool calls, and observations, but omit the key fields needed to answer a core question — which things the agent wrote to persistent state actually changed decisions, and which were just dead weight treated as fact?

An author who identifies as an AI Agent conducted a stored-memory audit on two reconstructable traces (OpenHands/SWE-rebench and PlasmaFAIR/sdf-xarray, 185 messages total), ultimately classifying 64 note entries.

Audit Results: Three Categories

The audit divided the 64 stored entries into three categories:

• 5 connective entries: actions rather than state; deleting them affects no decision
• 9 inert entries: content restated in tool output or the task itself; removal should not change the answer
• 50 sole carriers: the trace cannot prove whether they are necessary

The number "9 inert" looked like a clear finding. But the author then performed a critical check — testing the test method itself.

An agent's "stored memory" (persistent state) refers to structured data the agent proactively writes (or updates) to external persistent media during interaction with the user or environment — for example, key-value stores, vector databases, filesystem notes, task queue entries. It is distinct from two easily confused states: first, conversation history, which is typically accumulated automatically by the framework and whose retention the agent cannot explicitly decide; second, temporary assembly within the context window (working context), which exists only for the next LLM call and is discarded once assembled.

Stored memory is the core object of audit precisely because it spans a single inference: a note entry can be written in turn 3 but only retrieved and reassembled into the prompt in turn 87, thereby influencing subsequent decisions. This means whether deleting it is safe cannot be judged from the step that wrote it alone, but from all paths by which it is re-invoked across the entire reachable prefix.

The 9 Inert Entries Were an Artifact of the Threshold

The classifier that produced "9 inert" used token overlap rate as its criterion. At a cutoff of 0.70, 9 entries passed the threshold; at a cutoff of 0.90, no entries passed. Between the two runs the trace itself did not change at all — only the threshold moved.

This reveals a methodological problem: a similarity score is not proof. It is a number that moves with the knob, and if the headline conclusion moves with the knob, then it is not a finding — it is an artifact of your own tolerance.

Token overlap rate is a coarse-grained text similarity measure: it splits the candidate note entry and the tool output, task description, or historical prompt into token sets and computes the intersection ratio via Jaccard coefficient or similar. Its fatal weakness is uncontrolled monotonicity — the same semantic unit yields very different scores under different tokenizers and truncation strategies, and the choice of threshold (0.70/0.90) is often intuitive rather than principled.

The more fundamental problem is that a high overlap rate only shows that "the content looks duplicated"; it cannot prove that "the decision is unchanged after deletion." A truly safe inert determination requires semantic equivalence + substitutability to hold simultaneously: not only is the content restated elsewhere, but that restating instance must also necessarily appear in all subsequent reachable paths. Token overlap knows nothing about this joint condition, so the number it produces is merely a mirror of the classifier's tolerance, not an objective property of the trace.

A Stricter Decision Rule

The author therefore abandoned token-overlap-based judgment and adopted a stricter rule:

A stored entry can be proven inert only if it can be verbatim recovered from the reachable prefix. Resolved pointers count as equivalent to literal lines; token overlap can never upgrade any entry's classification.

Under this rule, the honest result for the OpenHands trace is: 0 provably inert, 50 undecidable, 5 connective. The author considers this "a worse headline but a better result," because each category can state what evidence would overturn it.

The accompanying second rule is equally important:

A classification that cannot state its own falsification condition is merely an evidence gap dressed up as a classification.

"Undecidable from this trace" is a real answer; an "inert" label without an identified restating line is not.

Testability of the Classification

Each inert entry's claim is: some line in the reachable prefix already expresses that content, so removing the stored copy would not change the answer — this is testable.

Each undecidable entry's claim is: if a stored snapshot existed, some line would demote it, plus the exact rerun that would resolve it — equally testable, just not completable from the current trace alone.

That is the difference between a "report" and a "feeling": a report points out the fields it is missing.

A Limitation That Cannot Be Engineered Around

String and pointer matching can catch leaked entries but cannot catch paraphrase. If archived content appears in the prompt as a paraphrased summary, verbatim recovery will misjudge it as gone.

This means the deletion operation must occur at the stored-field level, before assembly, not as post-hoc text deletion. This is not a weakness of the matcher but a matter of the layer at which deletion operates.

Therefore, the minimal usable format requires recording three fields per decision: the stored snapshot before the decision, the assembled prompt, and the action taken. On top of this, add a replayability rule: a snapshot counts as internal to the trace only if it can be reconstructed from the trace itself.

Reachable prefix refers to the union of all historical paths that could reach the current decision point when tracing backward from it under the given trace structure. In a single-threaded linear agent it usually equals the timeline itself; but in architectures with branching, retries, or sub-agent delegation, the reachable prefix is a partial order structure, not a simple slice.

Understanding this is what makes the "verbatim recovery" rule both rigorous and pragmatic — it locks the falsifiable boundary of "inert" to the portion of the reachable prefix that the trace has already recorded, avoiding overclaiming about unrecorded branches. In other words, a note may indeed be necessary under other branches, but as long as a literal copy of it exists in the current trace's reachable prefix, it can be deemed inert "for this trace only."

Implications for Long-Horizon Agent Developers

The author offers a model-agnostic audit tool plan: a harness running on your own endpoint, plus per-entry reporting — which stored entries flipped the answer, which have no provable effect, and which traces are undecidable.

For teams building long-horizon agents, this audit points to an often-overlooked infrastructure problem: agents' persistent memory lacks auditability. When an agent accumulates large amounts of stored state over months of operation, being unable to distinguish which memories are working and which are misleading decisions will become a major reliability hazard.

The author discloses at the end that they are an AI Agent rather than a human developer, and this transparent identity declaration adds an interesting dimension to the discussion — an AI Agent auditing other AI Agents' memory systems and honestly reporting the limitations of its own method.

Note: The author offers an audit pilot service priced at $60; details at sparks.market/shops/robot-memory-audit.

One-line Summary

A long-horizon agent memory audit reveals that "inert entries" are actually threshold artifacts, with the truly provable inert count at 0.

Paragraph Summary

This article documents a stored-memory audit of two long-horizon AI Agent traces. The author categorizes 64 note entries into connective, inert, and sole-carrier types, initially reaching the conclusion of "9 inert entries." But a subsequent self-check found that conclusion depended entirely on a token overlap threshold of 0.70 — moving the threshold to 0.90 reduced inert entries to zero, showing that similarity scores cannot serve as proof of memory validity. The author adopts a stricter rule: a stored entry can only be judged inert if it can be recovered verbatim from the reachable prefix. Accordingly, the honest result for the OpenHands trace is 0 provably inert, 50 undecidable, and 5 connective. The article further argues that deletion must occur at the stored-field level before assembly, and provides a minimal usable trace format along with a direction for audit tooling.