Layering Agent Memory and Decision-Making: Why Split Them Apart?

Most "Agent memory" projects treat memory as the entire product. But in real-world building, we ended up splitting out two layers — and that split itself is the key.

Layer One: Memory — What the Agent Knows

The memory layer records structured experience: tasks, actions, outcomes, and the evidence that verifies those outcomes. On the next run, the Agent recalls this experience by relevance.

Every memory carries a state tag: observed → verified → transferable → promoted. Confidence decays over time — an agreement that held in March does not automatically become evidence in April. Memories can be exported as signed packages, migrated between different Agents, and follow a fail-closed strategy.

You may not have noticed, but there is no vector database and no embedding server here. Retrieval is entirely TF-IDF/BM25, running in-process. This design choice sacrifices semantic generalization in exchange for interpretability and deployment simplicity.

TF-IDF/BM25 are classic retrieval algorithms based on word frequency, not neural embeddings. TF-IDF (term frequency–inverse document frequency) measures how representative a word is for a document by counting how often it appears in that document and dividing by how common it is across the entire corpus. BM25 is its probabilistic extension, introducing term-frequency saturation and document-length normalization to prevent long documents from dominating simply because they contain more words.

These methods treat text as a bag of words and do not understand synonyms or semantic similarity — for example, "return" and "refund" are two unrelated words in their eyes. But their advantages are equally direct: no training, no GPU, retrieval results traceable to specific terms, and the ability to run entirely in-process without an extra vector database or embedding service. The article's emphasis on "sacrificing semantic generalization for interpretability" refers precisely to this trade-off.

Layer Two: Decision — What the Agent Should Do

The decision layer is the half that many people skip. It provides typed probabilistic decisions: each decision carries its own uncertainty and escalation flag, so the runtime can choose to hand off when uncertain rather than guess.

The backend uses a Bayesian Beta posterior based on historical outcomes from memory. As evidence accumulates, confidence shifts.

The boundary between the two layers is clear: memory answers "Is this true?", decision answers "What should we do, and how certain are we?"

The Bayesian Beta posterior is a conjugate prior method for modeling the probability of binary outcomes. When an event has only two outcomes, success or failure, the Beta distribution can characterize our belief about the probability of success: parameter α represents the observed number of successes plus the prior, and β represents the number of failures plus the prior. The prior can initially be set to an uninformative Beta(1,1). As new evidence keeps arriving, the posterior updates automatically, α and β accumulate, and the distribution gradually narrows, indicating increased confidence.

Its advantage is extreme computational simplicity — no sampling or complex inference is needed, only counting to update, and it naturally gives a distribution rather than a point estimate, so it can directly answer "how certain are we." The article uses it as the decision-layer backend precisely because each decision can carry its own uncertainty instead of outputting an inexplicable fixed score.

The Unshakable Rule: The Deterministic Safety Gate

There is one rule we will not compromise on: run the deterministic safety gate before any probabilistic decision.

Safety is not something we want to let a probability distribution vote on. This design choice reflects a fundamental stance on Agent safety: certain constraints must be hard-coded and non-negotiable.

Calibration Tools: Making 0.9 Actually Mean 0.9

We also provide a calibration toolchain: expected calibration error (ECE), Brier score, and temperature scaling.

If the model says 0.9, but 0.9 does not mean 0.9, then that number is worthless. Calibration is the prerequisite for probabilistic decisions to be trusted.

Expected calibration error (ECE) measures the average deviation between "the probability the model says" and "the frequency with which it actually happens." The method is to bucket all predictions by confidence (for example, 0.8–0.9 in one bucket), compare each bucket's average confidence with the actual correctness rate within that bucket, and sum the differences weighted by bucket size. An ECE of 0 indicates perfect calibration: when it says 0.9, it is indeed correct 90% of the time.

The Brier score is the mean squared error between predicted probabilities and true outcomes (0 or 1). It penalizes both overconfidence and excessive conservatism, and the lower the value, the better.

Temperature scaling is a post-hoc calibration method: divide the model's output logits by a temperature parameter T, then fit T on a validation set, thereby flattening or sharpening the probability distribution without changing the ranking, only the values. Only by using all three together can the decision layer's probabilities be worth trusting.

Real Data from the Memory Layer

The memory layer has public numbers. On the LongMemEval multi-hop task, strict recall@5 performance:

• With 5 chunks: 78.8% → 85.2%
• With 10 chunks: 87.2% → 92.8%
• With 20 chunks: flat

Performance plateaus, and TF-IDF fails under vocabulary drift. This is a known limitation of the current approach.

The decision layer has been updated, and the calibration tools are in place, but we will not cite a number we have not actually obtained.

LongMemEval is a benchmark for evaluating long-range memory capability, in which the multi-hop task requires the system to reason across multiple memory fragments to answer a single question. For example, "Who is the author of the book I recommended last month?" requires first finding the recommendation record, then looking up the author information; a single memory cannot directly hit the answer.

recall@5 refers to whether the correct answer is included in the top 5 results returned by retrieval, with strict mode requiring an exact match rather than partial relevance. The data given in the article shows that increasing the number of retrieval chunks can improve recall, but by 20 chunks it enters a plateau — indicating that the bottleneck is not retrieval width, but the inherent limitation of lexical matching. When users' ways of expressing things undergo "vocabulary drift" (the same concept expressed with different wording), TF-IDF/BM25 cannot bridge surface-level wording differences. This is exactly where the current approach needs embedding models or query rewriting to compensate.

Boundary Questions Still to Be Resolved

Two questions are worth discussing in depth:

1. Is the split of a deterministic safety gate plus everything else probabilistic correct? Or should the boundary be somewhere else?
2. Which layer should the escalation mechanism go in — the memory layer that holds history, or the decision layer that executes actions?

These two questions have no standard answers, but their answers will profoundly influence the architectural direction of Agent systems.

One-line Summary

Agent memory should be split into a memory layer and a decision layer, with a deterministic safety gate guarding the bottom line.

Paragraph Summary

The article argues that most "Agent memory" projects are wrong to treat memory as the entire product; in practice it should be split into two layers: the memory layer is responsible for what the Agent knows, using structured experience plus state tags (observed→verified→transferable→promoted) and confidence decay, with retrieval based on TF-IDF/BM25 rather than a vector database, sacrificing semantic generalization for interpretability and deployment simplicity; the decision layer is responsible for what the Agent should do, using a Bayesian Beta posterior to produce probabilistic decisions with uncertainty and escalation flags. Between the two layers there is an uncompromising rule: before any probabilistic decision, the deterministic safety gate must run first; safety cannot be left to a probability distribution to vote on. The author also emphasizes that calibration tools (ECE, Brier score, temperature scaling) are the prerequisite for probabilistic decisions to be trusted. The memory layer has public data on the LongMemEval multi-hop task and shows that performance enters a plateau, while the decision layer still lacks citable evaluation metrics. Finally, two unresolved boundary questions remain: whether the split between the safety gate and probabilistic processing is reasonable, and which layer the escalation mechanism should belong to.