The Five-Layer Model of Agent Memory Architecture: Comparing Mainstream Framework Implementations

Title

Agent Memory Architecture's Five-Layer Model: A Comparison of Mainstream Framework Implementations

Meta Description

An analysis of the five-layer model of Agent memory architecture: history, compression, semantic recall, working memory, and file system. It compares the implementation differences among Mastra, LangGraph, OpenAI, and Claude to help developers choose memory solutions based on their scenarios.

Article Content

Agent memory capability is becoming the key differentiator between product-grade applications and demos. A technical analysis breaks Agent memory down into five layers: history, compression, semantic recall, working memory, and file system, and compares the implementation paths of Mastra, LangGraph, OpenAI, and Claude.

Why Agent Memory Needs to Be Understood in Layers

Treating Agent memory as a single feature is a common misconception. In reality, from conversation history to long-term knowledge accumulation, different needs correspond to completely different technical solutions. The value of the five-layer model is that it lets developers precisely determine which layer their scenario needs, rather than blindly piling on vector databases.

These five layers go from shallow to deep: the most basic is raw history records, followed by context compression, semantic recall, working memory, and the outermost layer is the file system. Each layer solves problems at a different granularity and brings different engineering complexity.

Layer 1: History — The Starting Point for All Frameworks

History is the most intuitive form of memory: store conversation messages in order and send them all to the model on the next request. Both Mastra and LangGraph adopt this pattern as their default behavior.

The problem lies in cost and window limits. As conversation turns increase, raw history quickly blows up the context window, and token costs grow linearly. Therefore, the history layer is usually only a starting point and needs to be combined with subsequent layers. OpenAI's Assistants API likewise stores message history in the form of threads, which essentially belongs to this layer.

Layer 2: Compression — Trading Space for Summaries

The compression layer solves the problem of history bloat. A common approach is to generate summaries of earlier conversations, retaining key information while discarding redundant details. LangGraph supports state trimming through its checkpoint mechanism, and developers can customize compression strategies.

Claude's implementation relies more on its long-context capability, using prompt engineering to guide the model to focus on key parts of the history itself, rather than explicit compression. These two routes represent different trade-offs: explicit compression saves tokens but may lose details, while implicit reliance on a long window costs more.

Context windows and token costs: A large model's context window refers to the maximum number of tokens that can fit in a single request (e.g., 128K, 200K). Historical messages, system prompts, and tool returns all occupy the window; exceeding it requires truncation or results in an error. Tokens are billed separately for input and output, and the cumulative cost of long conversations rises linearly with turns. The essence of compression is to retain high-information-density content within a limited window. Common strategies include rolling summaries, key entity extraction, and importance-scored trimming, with the goal of balancing cost and information retention.

Layer 3: Semantic Recall — The Home Turf of Vector Retrieval

Semantic recall lets an Agent retrieve relevant fragments from massive amounts of history, usually implemented with the help of a vector database. Mastra has a built-in semantic memory module that supports embedding conversations and retrieving them by similarity.

The engineering difficulty of this layer is not retrieval itself, but judging when to write — what content is worth remembering and when to trigger a write. According to the original analysis, most frameworks leave this part to developers to decide, lacking out-of-the-box strategies. This is also the part most prone to problems in current Agent memory systems.

Vector databases and embedding retrieval: Semantic recall relies on vector databases (such as Pinecone, Weaviate, Chroma) to map text into high-dimensional vectors and find semantically similar content through cosine similarity or inner product computation. Embedding models (such as OpenAI's text-embedding-3 or the open-source BGE) are responsible for encoding conversation fragments into vectors. From an engineering perspective, attention must be paid to chunking strategies, index updates, and the trade-off in retrieval top-k: chunks that are too large contain more noise, while chunks that are too small have incomplete semantics; untimely index updates will prevent new memories from being recalled.

Layer 4: Working Memory — The Scratchpad for the Current Task

Working memory refers to the intermediate state an Agent maintains while executing a single task, such as completed steps and pending subgoals. LangGraph's graph state mechanism is naturally suited to carrying working memory; the state passed between nodes is the concrete form of working memory.

Unlike semantic recall, working memory is short-term and within-task, and can be discarded once the task ends. Distinguishing these two is crucial for architecture design — mistakenly storing working memory in a long-term vector store will pollute recall quality.

LangGraph's graph state mechanism: LangGraph models the Agent workflow as a directed graph, where nodes represent processing steps and edges define transition conditions. Graph state is a shared data structure that runs through the execution cycle, and each node can read and update it. This design naturally carries working memory: completed steps, intermediate results, and pending subgoals are all stored in the state, and passing between nodes is the flow of working memory. After the task ends, the state can be discarded or persisted through checkpoints for recovery, thereby being clearly isolated from long-term memory.

Layer 5: File System — The Most Underestimated Memory Layer

The file system is often overlooked as a memory layer, but it provides structured, persistent, human-intervenable storage. Claude's code execution environment allows the Agent to read and write files, which in effect constitutes a reliable form of long-term memory. OpenAI's Code Interpreter is similar.

The advantage of this layer is transparency: developers can directly view and modify what the Agent remembers, without facing a black-box vector. For scenarios requiring auditing or debugging, the value of the file system far exceeds that of vector retrieval.

Framework Comparison: No All-Rounder

A horizontal comparison shows clear differences in coverage among frameworks. Mastra is relatively complete in semantic recall, LangGraph is stronger in working memory and state management, while OpenAI and Claude cover the history and file layers through their respective platform capabilities. No framework covers all five layers, and developers often need to combine them.

The core point of the original article bears repeating: Agent memory is not a single technology choice but a layered design problem. Clarifying which layers a scenario needs first and then choosing the corresponding framework capabilities is more reliable than directly applying some "memory solution."

Practical Recommendations

Start from the minimum viable approach: first get the process working with history records, add compression when hitting window bottlenecks, and introduce semantic recall when cross-session knowledge is needed. Working memory and the file system should be filled in progressively according to task complexity. Introducing a vector database too early often brings maintenance burden with limited benefit.

One-line Summary

Agent memory should be designed in five layers; no single framework covers everything, so combinations must be chosen by scenario.

Paragraph Summary

The article proposes a five-layer model of Agent memory: history, compression, semantic recall, working memory, and file system, solving problems at different granularities from shallow to deep. History is the starting point for all frameworks but is limited by window and cost. The compression layer trades space for summaries: LangGraph trims via checkpoints, while Claude relies on long-context implicit handling. Semantic recall uses vector retrieval, with the difficulty lying in judging when to write, and most frameworks provide no out-of-the-box strategy. Working memory is a within-task scratchpad, naturally suited to LangGraph's graph state. The file system provides transparent, auditable long-term storage and is often underestimated. The framework comparison shows Mastra is strong in semantic recall, LangGraph in state management, and OpenAI and Claude cover the history and file layers — there is no all-rounder. Practical recommendations start from the minimum viable approach and fill in layers as needed.