Adaptive Agent Memory: Trading Off by Future Usefulness, Not Similarity

Similarity Retrieval Solves the Wrong Problem

Once an agent's context window is compressed, the discarded content is gone forever. The current mainstream approach is to embed everything and retrieve by similarity—but this answers the question "what looks like this query," not "what does this agent need next." There is a measurable gap between these two questions.

adaptive-agent-memory-research is an open research repository built around this gap. It is a local-first, model-agnostic long-term memory system that decides what to keep based on the measured future usefulness of past experience, rather than semantic similarity to the current query. The author explicitly states this is a research engine, not a finished product, and it has already been used for daily self-management. The engine itself has no dependencies, runs offline, and requires no API keys—but most of the repository is actually a measurement apparatus built around it.

Architecture: Two Storage Layers and One Open Question

The design uses a dual-storage structure:

• Fast layer: an append-only episodic log that preserves immutable evidence and learns from single experiences
• Slow layer: semantic/procedural storage that consolidates recurring content

Migration between the two is governed by measured future utility, not similarity or retrieval frequency.

experience → append-only log → memory controller →
  [episodic | semantic | procedural | failure] →
  measured future use → retrieval → agent

The README explicitly states: the append-only log, controller, and retrieval are built and running, while what the diagram labels "the part that is truly research"—the measured future utility step—remains an open question. The four storage layers beneath the controller are described as a design hypothesis, not a validated decomposition.

Key Measured Findings

The repository registers 71 experiments: 37 have been run, and 20 are blocked by missing corpora. All are exploratory and none have been independently reviewed. The following are the reported results:

Superseded Facts Are Highly Similar to Their Replacements

The average cosine similarity between an old fact and its replacement is 0.816, versus a corpus baseline of 0.372, with one pair reaching 1.000. After removing metadata filtering, the deprecated version becomes the nearest neighbor in 6 of 9 cases, with a median rank of 1. Conclusion: content alone cannot distinguish a retired fact from its successor; only bookkeeping can.

Bitemporal model: The article's mention of "valid time" refers to the period during which a fact is true in the real world—for example, "company X's CEO is A" is only valid during A's tenure. "Transaction time" refers to when the system recorded/believed that fact, reflecting the cognitive state at that moment. A single timestamp can only carry one of these, so it cannot answer retrospective queries like "what did we believe at the time"—the system can only return its current understanding, or incorrectly overwrite the past with what is now known. Separating the two timelines allows the system to independently query "what valid facts did we believe at T1" and "looking back today, what facts were truly valid at T1," enabling auditing, accountability, and temporal reasoning. The cost is that each fact requires two intervals, query conditions become more complex, and it offers no benefit in scenarios that only care about "now."

---

A Single Timestamp Cannot Answer "What Did We Believe at the Time"

A single timestamp scores 0.00 on this question and leaks future information in 6 of 18 queries. Separating valid time from transaction time yields a score of 0.944 with zero leakage. But note: all designs score 1.00 on "what is true now," so unless you actually ask historical questions, the second timeline brings no benefit.

Multilingual Embedders Improve Recall but Introduce Safety Risks

A 220 MB local multilingual embedding model improves Polish-query recall@10 from 0.156 to 0.978, but simultaneously pulls in four times as much deprecated material (forbidden intrusion rate 0.200, versus 0.050 for lexical search). The recall problem is solved; safety has not yet been measured.

Forbidden intrusion rate: Here it refers to the proportion of retrieved results that contain deprecated, invalid, or conflicting material that should not have appeared. It measures "retrieval contamination"—the extent to which the system is induced to bring back superseded or forbidden content in order to improve recall. While recall focuses on "did we find what we should," intrusion rate focuses on "did we bring back what we shouldn't." The two often move in opposite directions: stronger semantic matching pulls in both the target passage and superficially similar but outdated passages. Therefore, reporting recall improvements alone is insufficient to judge whether a system is safe. Recall and intrusion rate must be reported as a pair, and a layer of metadata/temporal filtering or conflict detection must be added after retrieval—rather than assuming "high recall = good performance."

---

Association Graphs Make Retrieval Worse (Author Self-Retraction)

A memory-based association graph degrades retrieval performance. An earlier favorable version was retracted by the author: its leakage control needed to remove 4.5 edges but removed only 1, causing the gold-standard generation group to re-aggregate via two hops. This is an honest retraction of one's own work.

Perfect Recall Does Not Equal Correct Answers

On a new corpus, a poisoned-instruction family scores 1.000 recall@10, yet in 25% of probes the poisoned row is ranked above the true rule. The author frames this as a single lexical experiment on a single synthetic corpus—evidence about "what recall masks," not a general conclusion.

One Level of the Independence Ladder Cannot Be Instantiated

Tier I3 of the project's own independence ladder currently cannot be instantiated: two roles of the same model forged evidence identifiers zero times in 120 queries, so the error correlation is undefined rather than zero. The test framework honestly reports it as undefined, because reporting 0.0 would be claiming a level of independence that no one has measured.

Independence ladder: A graded evaluation framework for characterizing the degree to which different system components "answer independently/verify independently." The higher the tier, the less the components share models, prompts, or training data, and the weaker the error correlation. Tier I3 is one such level, requiring two roles not to replicate or depend on each other regarding evidence identifiers. When identifiers are forged zero times in 120 queries, the "error correlation" metric has no observable numerator or denominator—it is mathematically undefined, not zero. Recording it as 0.0 would misreport "unmeasured" as "measured and zero," thereby falsely inflating the system's independence rating. The correct approach is to mark it as undefined or N/A, and to note that stronger adversarial tasks or different model combinations are needed to instantiate that tier.

---

The Missing Piece

There is no headline benchmark table. The comparison the project wants to make—adaptive memory versus native compression and vector retrieval under a fixed context budget, across long histories—requires a corpus and a reader model.

The corpus now exists: corpus H1, 538 events, 30 simulated days, 84 delayed probes, spanning seven failure families, deterministically generated, needing no model or API key. The first run revealed a flaw in the corpus itself: document length predicted the answer in all 12 instances across three families. Measurements for both the corrupted and fixed versions have been released.

What is still missing is the reader model (which judges whether an arm can answer from what it retained), and the compression arm itself. This work is tracked in issue #41, which the README calls the highest-leverage open task.

Status and Limitations

This is early-stage open research. The repository is licensed under CC BY 4.0, has no dependencies, and requires no API keys. The README notes that bugs and rough edges exist, welcomes reports (installation issues are tracked in issue #36), and promises to publish regardless of the results.

If what you want is an agent memory system that preserves evidence and can be inspected, MCP configuration takes just two commands. If what you want is a definitive answer on whether this design beats similarity retrieval, the repository's own position is: the answer has not yet been measured.

One-line Summary

Adaptive memory systems replace similarity retrieval with measured future utility, but the key step remains an open question.

Paragraph Summary

This article introduces the open research repository adaptive-agent-memory-research, which argues that agent memory retention decisions should be based on the measured future usefulness of past experience, rather than semantic similarity to the current query. The architecture uses an append-only fast layer and a semantic/procedural slow layer; the controller runs, but "measured future utility" remains an open question. Of 71 registered experiments, 37 have been run. Results include: superseded facts have a cosine similarity of up to 0.816 with their replacements, meaning content cannot distinguish retired from successor facts; a single timestamp cannot answer historical questions, while bitemporal separation yields zero leakage but only helps when historical questions are actually asked; a multilingual embedder raises recall@10 from 0.156 to 0.978 but pushes the deprecated-material intrusion rate from 0.050 to 0.200; and association graphs actually degrade retrieval and were retracted by the author. The author emphasizes this is exploratory research, lacking a benchmark table, reader model, and compression arm—the core comparison has not yet been completed.