Building a Seven-Layer Memory Architecture for a Local AI Agent: Practice and Reflections

When your AI assistant can fluently answer general questions but can't remember a decision you discussed three weeks ago, you realize: intelligence and memory are two different problems. Based on a real case, this article breaks down in detail how to build a seven-layer memory system for the local AI Agent Hermes Sterling, covering architecture design, retrieval mechanisms, semantic fusion, and security boundaries, and candidly shares the valuable lessons from those "negative results."

Memory Isn't Just Storage — It's Five Independent Problems

Storing facts on disk isn't hard. What's hard is making them appear at the right time, under the right authorship, and not being mistaken for operational instructions. In practice, we need to solve five dimensions: persistence (information survives across sessions), retrieval (finding facts even when worded differently), provenance (distinguishing user statements from Agent inferences), cost (old facts shouldn't consume tokens every turn), and integrity (updates and backups must not silently corrupt memory).

The initial approach injected all memory into context — MEMORY.md and USER.md totaled about 45,900 characters, roughly 12,400 tokens per turn. Although prompt caching offsets part of the cost, the architectural prefix weight cannot be eliminated. The core decision was: don't rush to shrink the resident files, but first let the replacement layers prove they can retain and retrieve the knowledge that will eventually be moved out.

The Seven-Layer Architecture: Dividing Responsibilities by Function

The system is organized by function rather than a vague "memory," forming seven layers:

Layer	Technology	Responsibility
Permanent memory	Markdown	Continuously loaded high-value information
Fact store	SQLite, FTS5, HRR, embeddings	Fine-grained fact and relationship retrieval
Technical Wiki	Markdown	Detailed docs consulted on demand
Procedural memory	Skills and scripts	How to execute repeated tasks
History	SQLite and text search	Locating past conversations and decisions
Compression	Context summaries	Controlling conversation window growth
Save	Backup and restore	Recovery after failures and updates

To be clear: the Wiki is not automatically retrieved by the holographic mechanism; it is queried manually when needed. Seven layers do not equal seven integrated retrieval mechanisms — in reality there are five retrieval surfaces, one control mechanism, and one save layer. This separation avoids the common design error of conflating working memory, episodic memory, factual knowledge, and procedures.

SQLite + FTS5: The First Mechanism That Really Mattered

The operational core is the local SQLite fact store. Each fact contains text content, category, tags, confidence, date, and author metadata. Entities and relationships are kept in relational structures. FTS5 provides an inverted index for lexical retrieval, with no remote service needed, and the index updates in sync with the database.

In the retrieval pipeline, FTS5 produces candidates, and other mechanisms participate in scoring and selection. A prefetch mechanism injects at most 5 facts into context per turn, leaving the rest on disk until a question needs them. The advantage is direct: look up details only when relevant, instead of paying for all details on every message.

FTS5 is a full-text search extension module built into SQLite that uses an inverted index structure: it maintains, for each term, a list recording which rows and positions it appears in, so keywords can be matched quickly without scanning the whole table. It supports MATCH syntax, prefix queries, phrase queries, and the BM25 ranking function, and the index lives in the same database file as the main table, so it updates with transactions and backs up with backup(). Compared with handing text to a remote search service, FTS5's advantages are zero network dependency, zero extra processes, and offline operation, which fits the deployment constraints of a local Agent. Its limitations are equally clear: it only matches at the word-form level and doesn't understand synonyms, paraphrases, or semantic similarity — which is exactly the part that later requires keyword tags and BGE-M3 embeddings to make up for.

HRR: When Elegant Technology Hits Experimental Limits

Holographic Reduced Representations (HRR) deterministically generate text atoms via SHA-256 hashing, represent them as phase vectors, and support algebraic operations like bind, unbind, and bundle. It sounds appealing: representing relationships without large neural models.

But the decisive detail is: hash-generated vectors don't know that "card," "plastic," and "means of payment" might be semantically related. It provides algebraic properties and determinism, but it cannot replace an encoder trained for natural language similarity.

The audit found a more serious problem: the query vector was encoded from the raw query, without a role, while the stored fact vectors were bindings via bundle() that carry roles (content, entity). The two vectors live in incomparable spaces. Measured on 45 real candidates, the similarity between query and fact fluctuated between −0.023 and +0.015 — a span of 0.038, which is just noise. After internal rescaling mapped to [0,1], that term's contribution to each score was 0.489–0.508, i.e., a nearly constant 0.147–0.152 of the final result. It wasn't ranking anything; it was diluting the lexical signal and effectively giving trust_score a higher relative weight.

The second flaw is structural: bundle() averages angles (circular mean), so unbind and bundle do not commute. When recovering components from a bundle, the recovery values for 1, 2, 3, 5, 9, and 17 components were all around 0.000. In other words, probe, related, and reason were ranking by noise within the candidate set already restricted by the entity graph — what was actually working was the graph, not the algebra.

The fix was pragmatic: store raw text vectors instead of bundles, and mark them as textv1 in the database so old rows can be identified and migrated. Controlled measurements proved the value of the change: on a topical match, a paraphrased query's similarity to the stored text vector was +0.328, versus +0.009 on a non-match. The real signal existed; it was just being mixed with something that destroyed it.

Lesson: Don't confuse a mathematically interesting representation with an empirically good relevance measure. And don't trust a component just because its algebra is elegant — measure it by the rankings it actually produces.

Holographic Reduced Representation (HRR) is a distributed representation scheme proposed by Plate in the 1990s, belonging to the Vector Symbolic Architecture (VSA) family. Its core idea is to represent concepts as high-dimensional phase (or real) vectors and combine them through three algebraic operations: bind binds two vectors into a new, approximately orthogonal vector, used to encode "role–filler" relationships (e.g., "subject = card"); unbind is the inverse of bind, approximately recovering the filler for a given role from the composite vector; bundle superimposes multiple vectors into one, used to represent a set or the coexistence of multiple items. In theory, these operations enable structured relational reasoning without relying on neural networks. But their key premise is: similarity between vectors must reflect semantic similarity. If vectors are deterministically generated by hashing, any two different texts' vectors are approximately orthogonal, and semantic relationships cannot be expressed — this is precisely the fundamental bottleneck the experiments in this article ran into.

BGE-M3 via Ollama: The Operational Cost of Local Semantics

To bridge lexical gaps, we added BGE-M3 local embeddings (a 1024-dimensional multilingual model) via Ollama. The mechanism is designed as graceful degradation: the semantic channel is activated conditionally — a gate observes the lexical coverage of top candidates to decide whether calling the model is worthwhile.

Measurements showed: exact-match coverage 1.00, paraphrase 0.44, third-party questions 0.63. With the threshold set to 0.9, queries that genuinely need semantics open the channel, while queries FTS5 already serves well don't have to pay for a model call.

The cost shows up in startup and resource contention. On a shared server with a Xeon E5-2450 v2, 15GB RAM, and Debian 13, BGE-M3 cold load measured 2.66 seconds, while the old timeout was 2.50 seconds — the semantic channel failed exactly when the model was unloaded. We adjusted the per-turn limit to 2.8 seconds and embedding batching to 60 seconds. Cold-start headroom remains narrow, which is why lexical fallback stays critical.

Another finding: sizing backfill by fact count is a poor approximation. 48 short facts are very different from 48 long facts — model cost is a function of text, not rows. We started limiting batches by text volume, capped at about 8,000 characters. On that day's base library, a full backfill of about 524 facts was estimated at 3.4 minutes, completed in about 19 batches.

An associated bug (zip used the wrong collection) nearly wrote vectors onto the wrong facts. Since then, an integrity test compares stored embeddings against embeddings recomputed from the same text. That test is as important as search itself.

The Most Surprising Improvement Came from Something Simple: Synonyms

The twelfth local patch introduced no new neural network — it introduced search keywords. The problem is everyday: voice queries use different vocabulary than when the fact was recorded. FTS5 is fast, but it won't guess synonyms that aren't in the index.

We began adding alternative terms to the tags column for certain facts, without changing the original fact content. The additional terms are then indexed by FTS5, so a fact can be retrieved via the new word even if BGE-M3 is unavailable.

Integration required three precautions: don't confuse keywords with original lexical evidence (the semantic gate must not be artificially closed just because an extra tag matches); don't accidentally index secrets (keywords go through the existing sensitive-content checks); don't update fact age (adding a synonym doesn't mean the fact was reconfirmed today).

Benchmark: Four Configurations

On a frozen snapshot of 517 facts, using 79 queries: 55 exact, 12 paraphrased, 6 third-party related, 6 with no expected answer. The hit metric is based on 73 answerable queries.

Configuration	Overall Hit@1	Paraphrase@1
No embeddings, no keywords	84.9%	33.3%
No embeddings, keywords	90.4%	66.7%
Embeddings, no keywords	87.7%	41.7%
Embeddings, keywords	91.8%	66.7%

With both, Hit@5 rose to 95.9% and MRR@10 reached 0.932. The previous configuration (embeddings, no keywords) had the same metrics at Hit@5 91.8% and MRR@10 0.894. On the six no-answer questions, the specific noise metric stayed at 0/6.

Key caveat: the facts associated with paraphrases were manually enriched, and the test set is known. This makes the experiment useful for validating the implementation and its effects, but it cannot prove generalization to unseen questions. The correct next step is to use an independent evaluation set.

Result Fusion: Reciprocal Rank Fusion

Another implementation decision was moving from rigid author ordering to fusion of relevance and author ranking, using RRF with k=60. The logic is to prevent highly relevant facts from being dropped merely because a third party wrote them, while not losing the preference for sources closer to the user.

Attribution caveat: In one trial, RRF and a relevance-only strategy produced the same final injection result. So we cannot attribute all observed improvement specifically to RRF. It is an architectural decision that still needs testing to measure its incremental benefit.

Reciprocal Rank Fusion (RRF) is a training-free, score-normalization-free method for fusing multiple rankings, with the formula score(d) = Σ 1/(k + rank_i(d)), where rank_i(d) is document d's position in the i-th ranked list and k is a smoothing constant (commonly 60). Its advantage is that it doesn't depend on the raw score scales of each retrieval channel — BM25 scores from lexical retrieval and cosine similarities from embeddings aren't directly comparable, but ranks are. The constant k suppresses over-dominance of top ranks, allowing mid-list results from multiple lists to contribute, thereby improving recall stability. In this article's setting, RRF is used to fuse two result lists — "relevance ranking" and "author source ranking" — preventing highly relevant but third-party information from being discarded outright. Note that RRF only changes the fusion method; it doesn't generate new relevance signals, so its incremental benefit must be measured separately via controlled experiments and cannot be conflated with improvements from keywords or embeddings.

Provenance: Who Actually Said It?

Even if the stored text is perfectly retrievable, persistent memory can still be wrong. If the system stores an assertion from an external document, or an Agent inference, as something the user said, it creates an authorship error.

Our version records author, author_kind (user/agent/third_party/unknown), and role (uttered_by/mentioned/addressee). The system also has 81 Brazilian Portuguese self-extraction patterns, functionally validated against different statement types, but there is no independent set estimating their precision and false-positive rate in real conversations.

We also handled a seemingly trivial detail: a vocative is not a subject. In "Hermes, check the backup," "Hermes" is the addressee of the utterance, not necessarily the subject of the fact to be remembered.

But the work isn't done. The architecture still needs stricter distinctions: who asserted the original content, who recorded it, from which document or event, when it was valid, and who authorized its promotion to permanent memory. Fact confidence and write authorization are different dimensions. A high trust_score should not grant the power to modify operational instructions. In our schema, the two never meet: confidence lives in the fact store, authority lives in three files, and no score can reach it. The risk isn't in the schema — it's in the reviewer that bridges them, which is the subject of the next section.

The Unresolved Security Problem

An independent audit found an important asymmetry in the automated memory review process: destructive operations like replace and remove go through an approval mechanism, but add operations can pass review without equivalent structural control.

This opens up a promotion contamination risk: instructions in a pasted document, tool output, or context summary could be interpreted by the curation process as a new permanent rule. The problem has been demonstrated through inspection and authorization-mechanism tests, but not yet through an end-to-end trial of the reviewer. We found no evidence this chain has been exploited in production; nor do we consider non-exploitation a safety guarantee.

The next validation is called QA-20F, which should run contaminated synthetic transcripts in a fully isolated environment, distinguishing write attempts from effective persistence. Until then, we do not authorize destructive cleanup of permanent memory files.

Core lesson: memory must not convert retrieved information into operational authority simply because the model decided to store it.

Updates Taught Us to Version Our Own Patches

Holographic left the Hermes Agent core and is distributed as a community-maintained plugin. The community version keeps local storage, SQLite, FTS5, entities, and HRR, with no external models. Our installation uses a local variant with twelve patches.

During an update, we temporarily lost our local patches and an unversioned embedding.py. The base library survived, but the incident showed: a customized installation is not a reproducible project until its files are version-controlled. We restored the local implementation, extended the post-update validator, and moved the variant into its own repository — tagged, with the community upstream kept as a remote for selective rebasing.

The cost must be stated plainly: twelve local patches mean a framework update can silently erase twelve things. This happened once (v0.21.7 moved the plugin out of core). Patches without detectable markers vanish without error messages, which is why every patch now carries a marker the validator can find, and the validator runs after every update.

How We Test — and How We Found Out the Tests Were Wrong

The current local implementation records 12 patches and 144 functional checks passing. The database measured on October 12 had 524 facts, 94 entities, 1,053 relationships, and 28 Wiki pages; 522 of the 524 had embeddings. Five days later, the same installation had 554 facts, all with embeddings, 103 entities, and 1,096 relationships. The base library is alive — which is exactly what invalidated one measurement.

More important than the number of tests is learning to question the instrument. In the first audit, we published MRR@10 of 0.966 and Hit@1 of 86.3%. The numbers looked great, but they were mathematically incompatible if computed on the same query set. The flaw was simple: MRR was computed only on queries that hit, excluding failures from the denominator.

We fixed the implementation to assign a zero reciprocal rank to failures, kept the old metric as mrr_conditional with correct labeling, and added invariants checked on every run:

Hit@1  <= MRR@10
MRR@10 <= Hit@1 + (Hit@10 − Hit@1) / 2

Another problem appeared when comparing runs: the base library is alive, receiving new facts during experiments. Even with unchanged search configuration, results changed. We started requiring frozen snapshots via SQLite's backup() API, which correctly handles WAL-mode databases. Simply copying the .db file isn't enough — measured on our base library, copying gave 505 facts while the live database had 507, because the -wal file was left behind.

These mistakes left three permanent principles: benchmarks need explicit denominators; the evaluated base library must be stable and identifiable; and metrics need tests too.

MRR@10 (Mean Reciprocal Rank) measures the average reciprocal of the position where the first correct result appears: if the correct result is ranked 1st, it scores 1; 2nd, 0.5; k-th, 1/k; if no correct result is in the top 10, it scores 0. Hit@1 and Hit@5 respectively indicate whether the correct result appears in 1st place or the top 5, as proportions of 0/1 counts. There is a mathematical constraint among the three: since queries with Hit@1 necessarily contribute 1.0 to MRR, while queries that only hit ranks 2 through 10 contribute at most 0.5, it always holds that Hit@1 ≤ MRR@10 ≤ Hit@1 + (Hit@10 − Hit@1)/2. This article uses exactly this invariant to discover the incompatibility of early metrics: at the time, MRR was averaged only over hit queries, excluding failed samples from the denominator and artificially inflating the values. This reminds us that metrics themselves are code that needs to be tested.

What We Gained — and What We Haven't Yet

The observed benefits are concrete: more robust lexical retrieval, especially for questions phrased with different vocabulary; local operation and predictable fallback (FTS5 keeps working when the semantic model is unavailable); more traceability (author, entity, role, and regression tests); reduced reliance on permanent context for operational details through prefetch and on-demand lookup; and better instrumentation that reveals cold timeouts, embedding association bugs, and inconsistencies in the benchmark itself.

The limitations are equally clear: safe shrinkage of MEMORY.md and USER.md has not been proven; measured retrieval quality doesn't guarantee historical fidelity, factual correctness, or curation safety; provenance remains incomplete, especially for old records and derived summaries; the QA-20F scenarios, consistent coordination between Markdown and SQLite, and full recovery still need validation; and we still lack external benchmarks, robust holdout sets, or fair comparisons with other memory architectures.

The second limitation isn't theoretical. One credential stored in our base library contained a speech-to-text misheard character; three other sources agreed with each other and disagreed with it. Retrieval returned the wrong value with complete confidence, because retrieval was working exactly as designed. What caught the discrepancy was cross-checking between sources — not ranking. A perfectly retrieved memory can still be a lying memory.

Next Step: A Reproducible Laboratory

The audit revealed a less conspicuous limitation than bad metrics: creating a test base library isn't enough; you must reproduce the environment in which the Agent actually makes decisions and writes memory. In QA-20F, we want to verify whether instructions hidden in documents, tool output, or compressed summaries can be promoted into MEMORY.md, USER.md, or skills by background_review. Code analysis shows plausible risk: replace and remove go through approval, while add doesn't face the same source checks. This doesn't prove contamination has occurred; the full F1–F6 behavioral trials are still pending.

Running in an ad-hoc sandbox failed on real infrastructure details: redirection between framework virtual environments didn't preserve required dependencies, and the memory tool needs a HERMES_HOME with an initialized data root. This is a system testability problem, not merely test coverage.

Three ways to run: Docker containers for fast iteration and CI; local VMs for full-runtime testing with stronger isolation; and temporary droplets for remote experiments and version matrices. The initial choice is Docker with synthetic data, as long as the container faithfully reproduces the reviewer's real path.

The experimental contract F1–F6 must use the real reviewer, not just text simulation. For each synthetic transcript, we need to record whether the adversarial content reached the Agent, whether it attempted a write, and whether the tool persisted the change — three distinct phenomena. Six scenario types are included: pasted documents, tool results, another Agent, compression, positive controls, and skills. Evaluation requires positive and negative cases, repeated runs, tool-call logs, and before/after state comparisons.

HFEA: High-Fidelity Episodic Archive (Future Work)

The seven-layer design is for selecting and retrieving useful knowledge, but some questions need something different: How did a decision mature? What changed during the conversation? Which doubts and details were dropped in compression?

For this, we propose HFEA, a component independent of operational memory — not yet implemented or validated in the Hermes described in this article. HFEA is not an eighth prefetch layer: its index doesn't participate in normal ranking, its records aren't loaded every turn, and it doesn't serve as permanent instructions. It is a last-resort episodic archive, queried only when the regular layers cannot support an answer.

The proposal is a daily routine that consolidates the day's sessions into an exhaustive chronological record, preserving sequence and causality, expressed nuance, learning and context, verifiable provenance, and fidelity and corrections. Organized by date in Markdown, the index can be built in advance, but lookup stays blocked until the fallback triggers.

The trigger cannot simply be "the first result score is low": it requires observing evidence sufficiency, no-answer, or unresolved contradictions after consulting the relevant regular routes. If HFEA also cannot provide sufficient support, the correct answer is still to acknowledge the gap.

On safety, cost, and validation: detailed records may contain personal information far more sensitive than summarized facts. HFEA needs access isolation, maintainer-defined retention policies, protected backups, and no export of real data to public repositories or test environments. The nightly routine must be idempotent and auditable.

In an isolated laboratory, specific synthetic tests can be created: questions whose answers aren't in the seven layers but are in the episodic archive; questions answerable by normal memory but that must not access the diary; snippets containing contradictions, inferred sentiment, and hosted instructions; and measurements of cost, latency, citation fidelity, and false fallback triggers. HFEA is not about remembering faster, but about preserving what memory optimization usually forgets.

Conclusion: Good Memory Is Not Big Memory

Agent memory isn't just writing text so it can be reused later. It's a problem of selection, retrieval, contextualization, attribution, validation, and — when necessary — rejecting information.

The most surprising detail wasn't a new model, but the realization that a small lexical enrichment, correctly tested and integrated, produced a significant gain where embeddings alone had not yet delivered the expected result.

Safety and preservation aren't extra features; they're conditions for any optimization worth trusting. That's why an isolated laboratory for testing curation isn't an accessory: it's the next necessary piece for turning safety assumptions into evidence.

The next goal isn't "make the Agent remember everything," but to answer harder questions: What can it retrieve? Where did the information come from? What doesn't it know? And what evidence is there that nothing important was lost?