Agent Memory Explained: The Four-Layer Architecture and the 'Dreaming' Mechanism

How do AI agents remember things between sessions? Anthropic's research article "The Components of Agent Memory" offers a systematic breakdown. Drawing on the interpretation by the YouTube channel My2Cents AI, this article walks through the definition of agent memory, its four-layer architecture, what should be stored, and a counterintuitive mechanism—letting agents "dream."

What Counts as Memory? Three Things That Don't

Anthropic defines agent memory as: knowledge that Claude acquires from experience and retains as readable context between sessions.

Three things are explicitly excluded: the model weights themselves, what is learned within a single conversation, and compaction—the mechanism that replaces earlier turns with a summary. These all belong to context, not cross-session memory.

This distinction matters. It means memory is not an extension of model capability, but an external knowledge system that is readable, writable, and auditable.

Compaction refers to the mechanism that, within a single session, replaces the raw content of earlier turns with a summary when the conversation history approaches the model's context window limit. It solves the problem of "can we keep talking right now." The compressed summary remains part of the current context, disappears when the session ends, and is not carried over to the next session. That is why Anthropic excludes it from agent memory: compaction is a context-management technique, whereas cross-session memory is an external knowledge system that is independent of model weights and can be persistently read and written. The two differ fundamentally in lifecycle and auditability.

The Four Layers of Memory

From bottom to top, memory is divided into four layers:

First layer: transcripts (working records). A searchable log of the agent's work—the raw material of all memory.

Second layer: memory notes. Notes that Claude writes for itself in the course of working.

Third layer: representations. Something like interlinked wiki pages that organize scattered notes into structure.

Fourth layer: injected context. Content that is automatically loaded into every new session—sitting at the very top.

The "dreaming" mechanism runs across the entire stack: it reads transcripts to correct notes, builds the wiki, and decides what to load next time.

Memory Is a Prediction

Anthropic's core idea is that memory can be a prediction—a bet that "this note will come in handy later."

They use the CLBench benchmark to test this betting ability. Each question runs in a fresh session; Claude then sees the correct answer, and only the memory folder is carried between sessions. The ideal loop is: record the failure → form a theory → test it → turn it into a rule.

Results diverged sharply. Sonnet 4.6 got stuck on unverified theories, making almost no progress. Opus 5 was the first of the tested models to close the loop, jumping from 0.56 to 0.83.

CLBench is Anthropic's benchmark for evaluating an agent's "memory judgment." Its key design isolates the memory variable: each question runs in a fresh session, the model cannot see prior conversation history, and the only thing passed between sessions is its memory folder. That way, changes in scores can only be attributed to what the model wrote into memory and what it read out next time. The ideal problem-solving loop is "record the failure → form a theory → test it → crystallize it into a rule," and CLBench measures precisely whether the model can close this loop—not its single-turn reasoning ability.

The Pokémon Experiment: What Is Worth Remembering?

The roughly ten-thousand-step Pokemon Red experiment turned "judgment" into an observable metric.

Sonnet 3.7 stored almost everything—one file for every battle, every item, every character. But the game state kept changing, and these files quickly went stale. At one point, it created 6 files and deleted 9 over 17 steps without performing a single in-game action.

Fable 5 stored what couldn't be seen on screen: goals, plans, and rules it had verified—for example, "switch timing after three failed Poké Balls in a row."

Opus 5.5 likewise kept a small folder and completed the game.

So should we just tell the model how to take notes? The article's advice: explain what memory is for, not how to use it. Rigid rules (such as "only record failures") cost strong models most of the benefit. But for older models, guidance helps the most—telling Sonnet 4.6 to read its memory and test theories nearly doubled its score. Sonnet 5 does this on its own, reading memory first nine times out of ten.

The Pokemon Red experiment is a long-horizon agent test environment that Anthropic used to observe memory "judgment," spanning roughly ten thousand steps. The game was chosen because it naturally has long-horizon goals, hidden state (such as the opponent's team and unexplored map areas), and plenty of quickly-stale local information, which forces trade-offs about what to remember and what not to. The experiment compared the size and content of memory folders across different models under the same framework: storing more is not necessarily better—files that go stale as the game state changes actually drag down performance—while a small folder recording only goals, plans, and verified rules often goes further and can complete the game.

The Five Jobs of Dreaming

Correcting wrong notes. In one run, Claude marked a landmark on the wrong side of Mt. Moon, writing "Mt. Moon, done." It wasn't. It then took a one-way path and fell off a ledge back to where it started. The dreaming mechanism compared this against the transcript, found the error, and rewrote the note; upon returning, Claude reached Cerulean City. Without the correction, it fell off the ledge and retreated to Pewter City three times; with no memory at all, it never arrived.

Building new structure. Claude "dreamed" a wiki out of roughly 350,000 internal documents from Stainless, which Anthropic acquired. Over 128 sessions, the Opus 5 agent wrote and linked pages, merged duplicates, and checked against sources. On test questions, the wiki alone improved from 13% to 48%; search alone held at 53%; wiki plus search reached 61%, with fewer tokens. But in legal tasks—where most answers concentrate in a single document—the wiki showed no clear gain over search alone. And one wiki grew larger than its sources—a map the size of its territory.

Promoting notes to the top layer so they load automatically into every session. The cost is tokens, and a single wrong note propagates to all sessions; the longer the instructions, the less strictly they are followed. So the top layer must stay small. Claude Code's auto-memory follows this principle: it loads only the first 200 lines and 25KB of memory.md, as an index of "what Claude knows and where to find it."

Aligning memory with the goal. In a legal task, under basic instructions, multiple agents wrote simultaneously to a shared table of contract clauses, erasing each other's rows, and the brief captured only 41 of 57 facts. Anthropic ran a loop: Claude reflected on the failure and rewrote the wiki-building instructions—one page per contract, terms quoted verbatim—and got all 57.

Filling gaps. A second scan found nine links pointing to non-existent pages, and they were re-pointed.

Practical Takeaways and Open Questions

The author distills a few principles: store what the environment won't tell you; keep memory curated and the top layer small; explain what memory is for; let the agent dream.

Open questions are equally real. Writing memory mid-task distracts attention, while dreaming's output only appears later; keeping memory current consumes compute. As the article puts it, no model can truly know what will happen in the next few rounds—dreaming lets it look back.

The original article is "The Components of Agent Memory" on the Anthropic website.

Background

Auto-memory is the mechanism in Claude Code that automatically injects part of memory into every session. It follows the "keep the top layer small" principle: it loads only the first 200 lines (~25KB) of memory.md, as an index of "what Claude knows and where to look," rather than stuffing all memory into context. This is a trade-off between recall and token cost—top-layer injection ensures key indexes are always visible, while details remain at lower layers for on-demand retrieval; limiting the size also reduces the risk of a single wrong note propagating to all sessions and of long instructions being ignored by the model.

One-line Summary

Anthropic breaks down the four-layer architecture of agent memory and its "dreaming" mechanism, revealing what to remember and how.

Paragraph Summary

Drawing on Anthropic's "The Components of Agent Memory" and My2Cents AI's interpretation, this article systematically examines agent memory. Anthropic defines it as readable context retained across sessions, excluding model weights, in-session learning, and compaction. Memory has four layers: transcripts, memory notes, representations, and injected context. The dreaming mechanism runs across the stack, correcting wrong notes, building the wiki, improving top-layer injection, aligning with goals, and filling gaps. CLBench shows that memory is a predictive bet, with Opus 5 the first to close the loop of "record failure → form theory → test → turn into rule." The Pokémon experiment shows that storing what the environment won't tell you (goals, plans, verified rules) beats exhaustive note-taking. The article recommends explaining memory's purpose rather than prescribing methods, and notes open questions such as token cost and error propagation.