Agent Memory Systems in Practice: Designing and Implementing Long-Term Memory Architecture

Why LLMs Are Inherently "Forgetful"

A common question among developers new to agent development is: why does the same LLM fail to "recognize" me in a second conversation? The answer is actually simple — LLMs have no true persistent memory; they only have "context."

Here's an intuitive example: suppose you make two consecutive calls to an LLM. In the first call, you tell it "I'm Lao Xiao." In the second call, if you don't repeat that statement, the model has absolutely no idea who you are. The only native way to make the model remember you is to pass "I'm Lao Xiao" back in with every single call.

But here's the problem: if you have to pass in all historical information every time, the content keeps piling up and will inevitably exceed the context window size limit. The context window is a core constraint parameter in LLM architecture — it determines the maximum number of tokens the model can process in a single inference. Tokens are the basic unit of text processing for LLMs; a single Chinese character typically corresponds to 1–2 tokens. Take GPT-4 Turbo as an example: its context window is 128K tokens, and Claude 3.5 supports 200K tokens. While these seem large, in real Agent scenarios, system prompts, tool descriptions, user inputs, conversation history, and retrieval results all share this window, leaving far less usable space than you might expect. More critically, longer context windows mean higher inference latency and greater API call costs. So even when the technology allows longer inputs, engineering considerations demand careful budgeting.

This is the fundamental reason Agent memory systems exist — models have no memory capability, yet we can't brute-force all history into the context.

The core objective of a memory system can be distilled into a single sentence: Inject the right memories into the limited context at the right time.

Context ≠ Memory: Clarifying Two Core Concepts

This is the second most common misconception among beginners: equating "context" with "memory." In reality, they are entirely different concepts.

What Is Context?

Context is the entirety of what you pass to the model in a single call. Think of it as documents on a meeting table — valid for one session, then discarded. The core question context answers is: what can the model "currently" see and know?

What Is Memory?

Memory is information that the system preserves long-term and can retrieve again in the future. Memory is a broad concept that typically includes:

• Knowledge bases
• User preferences
• User profiles
• Historical chat records

Context is really just a "one-time invocation form" of memory. Conversation history does originate from context — during a conversation, you send previous exchanges along to the model — but conversation history is by no means equivalent to context, because history keeps accumulating and will inevitably exceed the context window at some point.

Once you understand this distinction, you'll grasp what agent engineering is really about: building a dynamic scheduling mechanism between context and memory.

Three Pitfalls of Relying Solely on Chat History as Memory

Some might think: just store all chat records and inject them entirely into the context each time. This approach leads to three critical problems.

Pitfall 1: Context Explosion

Chat records grow continuously. Full injection will eventually breach the model's context length limit, directly causing call failures. This is the most straightforward technical bottleneck.

Pitfall 2: Attention Dilution

Even if the records haven't "blown up" the context yet, cramming large amounts of irrelevant content all at once dilutes the LLM's attention. The root cause lies in the Transformer architecture and its Self-Attention mechanism underlying LLMs. The essence of the attention mechanism is computing relevance weights between every token in the input sequence to determine what to "focus" on. When the context is stuffed with irrelevant content, attention weights get dispersed across this noise, reducing the model's focus on truly critical information. This phenomenon is known in academia as the "Lost in the Middle" problem — research shows that LLMs have significantly lower recall rates for information in the middle of the context compared to the beginning and end. Therefore, even within the allowed context window range, the quality and ordering of information directly impact output quality. Faced with a pile of redundant information, the model struggles to grasp the key points for the current task, and response quality degrades accordingly.

Pitfall 3: Memory Gaps

Simply stacking chat records also creates "memory gaps." For example, user preference information — without proper summarization and synthesis — simply cannot be extracted from raw chat records. When you need to perform user intent understanding and intent classification, having only scattered historical conversations without well-distilled user profiles severely undermines decision accuracy.

Conversely, if you provide user preferences and profiles alongside necessary chat history to the model before invocation, intent understanding and decision accuracy naturally improve.

A Better Agent Memory Architecture Design

The right approach is layered management + dynamic injection.

Separating Short-Term and Long-Term Memory

• Short-term memory: Stores the current session state, serving the current or recent continuous interactions.
• Long-term memory: Stores cross-session user preferences, facts, profiles, and other information requiring persistent storage.

The significance of this layering is that different types of information have different lifecycles — managing them together only creates confusion and performance issues. This design philosophy closely aligns with memory models in human cognitive science. Psychologists divide human memory into working memory (limited capacity, immediate processing) and long-term memory (nearly unlimited capacity, persistent storage). An Agent's short-term memory corresponds to working memory, while long-term memory corresponds to the human long-term memory system.

Dynamic Memory Injection Before Each Call

The key mechanism is: before each LLM call, retrieve relevant memories on demand from various sources and inject them into the context, keeping the context as concise as possible.

Think of the context as a "window," but the contents of this window are dynamically changing, not fixed. Each time the model is called, the system determines what the current task requires, then pulls useful content from short-term memory, long-term memory, knowledge bases, and other sources, assembling a refined context to pass to the model.

In the long-term memory retrieval process, vector databases (such as Pinecone, Milvus, Weaviate, Chroma, etc.) play a core infrastructure role. The working principle is: first, convert text into high-dimensional vectors using an Embedding model (such as OpenAI's text-embedding-3-small), then store them in a database optimized for Approximate Nearest Neighbor (ANN) search. When the Agent needs to retrieve memories, the current query is similarly converted to a vector, and semantically relevant memory fragments are found using algorithms like cosine similarity. This semantic similarity-based retrieval far outperforms traditional keyword matching because it can understand the semantic connection between "I don't want spicy food" and "user prefers mild flavors." This is also one of the core components of RAG (Retrieval-Augmented Generation) technology.

This way, the context stays concise (avoiding explosion and attention dilution) while ensuring the model gets the most relevant information each time (avoiding memory gaps).

Summarization and Compression Is a Non-Negotiable Step

Building long-term memory is inseparable from summarization and compression. Massive volumes of raw conversation must be summarized and refined to distill into reusable user preferences and profiles.

Memory summarization is typically performed by the LLM itself — in engineering, this is called "Recursive Summarization" or "Progressive Compression." Common implementation strategies include: sliding window summarization (when conversation turns exceed a threshold, generate summaries of earlier conversations to replace the original text), hierarchical summarization (first summarize each conversation segment, then produce a second-level summary of summaries), and entity extraction (extracting structured user preferences, facts, relationships, etc. from conversations into user profiles). For example, open-source memory management libraries like Mem0 implement automatic extraction of key memories from conversations, deduplication, merging, and conflict resolution. The essence of summarization and compression is replacing large volumes of low-density raw data with small amounts of high-density information — achieving "lossy compression" of information. This step is a critical, non-skippable part of memory system engineering.

Four Core Steps for Implementing a Memory System

In summary, an agent's memory capability is not a single technical point but a systems engineering effort:

1. Define storage scope: Determine which data needs to be stored in memory — this varies across projects;
2. Design a layered structure: Short-term and long-term memory each serve their own purpose without interfering with each other;
3. Establish a dynamic injection mechanism: Retrieve the right memories at the right time and inject them into the context;
4. Summarize and compress historical data: Synthesize and refine raw conversations to distill high-value information.

In practice, this architecture typically leverages orchestration frameworks like LangGraph to manage session state and workflows. LangGraph is an Agent orchestration framework from the LangChain team, designed specifically for building stateful, multi-step agent workflows. Unlike traditional chain-based invocations, LangGraph uses directed graphs to define Agent execution flows, where each node represents a processing step (such as memory retrieval, model invocation, or tool execution) and edges represent transition conditions between steps. Its core advantage is a built-in state management mechanism — developers can define information that needs to be passed across steps in the graph's State, including short-term memory and session context. LangGraph also supports a Checkpoint feature that persists session state to a database, enabling state recovery across requests, making it particularly well-suited for Agent scenarios requiring complex memory management.

Combined with vector databases for storing and retrieving long-term memory, the organic integration of context and memory is exactly what determines whether an agent product can truly "remember its users" and deliver a coherent experience. Understanding this underlying logic is the key to avoiding detours in engineering implementation.