LLM Agent Memory Systems Explained: From Buffer to Semantic Memory

Why Do Agents Need Memory?

The ultimate goal of building an AI Agent is to make it an orchestrator that can coordinate LLMs, invoke tools, and possess memory. Every individual LLM call is inherently "amnesiac"—the model doesn't remember what you asked in the previous turn; it simply generates a response based on the context passed in at that moment. To make a conversation coherent, the application layer must resend the historical Q&A back to the model.

The essence of memory is storing Query-Response pairs—"what I asked before, and what it answered"—and passing them back as needed in subsequent calls. But this immediately raises an engineering question: conversations can grow indefinitely, and we can't cram everything into the context. This gives rise to different memory strategies, each trading off between cost, context length, and information retention.

Context Window refers to the maximum total number of Tokens a model can receive in a single LLM call, including the system prompt, historical conversation, tool return results, and the current user input. Window sizes vary widely across models, from 4K in early days to the common 128K or even larger today. The larger the window, the more history you can fit—but attention and cost rise accordingly. More importantly, even with a sufficiently large window, the model's ability to recall information in the middle of a long context degrades—a phenomenon often called "Lost in the Middle." Therefore, memory strategy isn't just about whether information "fits"; it also needs to consider whether the model can actually "use" it once placed. When designing memory for an Agent, context window capacity should be treated as a limited budget, not a free resource you can overdraft indefinitely.

Breaking Down the Six Main Memory Patterns

The original article categorizes memory solutions into six types: Buffer, Window, Token, Summary, Long Term, and Semantic. Below we expand on each with implementation details from LangChain.

Buffer Memory: Full Storage, Simple but Expensive

Buffer memory stores the complete conversation for each chat, optionally in memory (RAM) or persistent storage (such as SQLite3 or PostgreSQL). Every call sends the entire history to the LLM, and a new conversation is assigned a new identifier.

At the code level, InMemorySaver() stores state in RAM, with capacity limited by available memory. The problem is straightforward: full history continuously consumes Tokens and pushes against the context limit, with cost and latency growing linearly with conversation turns.

Token is the basic unit of text that an LLM processes, roughly corresponding to a word or subword fragment. In English, one Token is roughly 0.75 words; in Chinese, depending on the tokenization method, one character may correspond to one or two Tokens. Billing, context length limits, and trimming budgets are all measured in Tokens, so "number of messages" and "number of Tokens" are not equivalent—one very long message may consume more budget than ten short ones. This is also the core difference between Token Memory and Window Memory: the former measures by actual consumption, the latter by message count. When estimating costs, developers should count Tokens using the tokenizer of the corresponding model, rather than simply converting by character count or message count.

Window Memory: Keep Only the Beginning and End

Window memory retains only a fixed number of messages, typically "first message + most recent N messages." LangChain provides a middleware mechanism where requests must pass through this layer before reaching the model, and developers can configure message-trimming logic there to keep only the beginning and end.

This strategy suits scenarios where "recent conversation matters more than early conversation," at the cost of completely forgetting the middle portion.

Token Memory: Trim by Budget

Token memory is similar to Window, but uses Token count rather than message count as the budget cap—for example, setting 2000 Tokens, with anything beyond that no longer retained. The definition of a Token varies by model and framework.

Here's an easily overlooked detail: trimming may cut a sentence in the middle, resulting in incomplete semantics. The original article notes that even when using exactly 1000 Tokens, to obtain complete semantics you often need to re-truncate from the start of a sentence. The trim_messages() function handles such edge cases. The hard flaw of this approach is that any old content exceeding the budget is completely lost.

Summary Memory: Trade Memory for Summaries

The idea behind Summary memory is to compress old messages into a summary before they are deleted, so that even if the raw data is gone, key facts remain. Summaries are generated with a cheaper small model to control costs, using the SummarizationMiddleware() function.

It incurs the overhead of an extra AI call, but in exchange retains long-term facts—suitable for assistant scenarios that need to remember key information across multiple turns.

Long Term Memory and Semantic Memory

Long-term memory lets the Agent itself decide "what to store and where," with facts saved in a Store that can be shared across multiple conversations by the same user. Semantic memory compresses an entire conversation into a point and stores it in a vector database via Embedding, retrieving by semantic similarity—essentially the same idea as RAG. The difference is that semantic memory requires an additional Embedding model.

Embedding is the process of mapping text into high-dimensional vectors, where semantically similar texts are closer together in vector space. Vector databases (such as FAISS, Pinecone, pgvector) use approximate nearest neighbor search to quickly find the records most similar to a query vector. In Semantic Memory, each conversation is embedded and stored in the vector database, and at retrieval time the current question is used to generate a query vector and recall relevant history. This pipeline is identical to RAG (Retrieval-Augmented Generation)—the only difference is that the knowledge source is "historical conversation" rather than "external documents." Introducing an Embedding model means additional inference cost and dimensionality management challenges, but it buys the ability to recall memory by semantics rather than by time or message count.

Key Concepts in Engineering Implementation

Once you understand the strategies, you also need to clarify a few foundational building blocks in the LangChain ecosystem, or confusion easily arises.

• Checkpointer: Saves state after each conversation turn; InMemorySaver stores in memory, while SqliteSaver or a database is used for persistence.
• thread_id: Essentially the "file number" of a conversation; the same id continues the same conversation, while a new id starts a brand-new one.
• Middleware (@before_model): Runs before each AI call and can edit the message list.
• Difference between Store and Checkpointer: Checkpointer manages a single conversation, while Store manages long-term user facts across conversations.
• Secure source of user_id: The user_id in memory tools should come from the application context rather than being generated by AI, to prevent users from reading others' memory beyond their authorization.

These concepts together form the skeleton of Agent memory: Checkpointer ensures single-session continuity, Store provides cross-session personalization, and Middleware handles context trimming before calls.

LangGraph is an orchestration framework in the LangChain ecosystem for building stateful Agents. It models an Agent's execution flow as a Graph, where nodes represent steps and edges represent transitions. Checkpointer and Store are both state management components of LangGraph: Checkpointer snapshots the graph state after each node execution, thereby supporting resumption from checkpoints and session recovery; Store is a key-value store independent of the graph, used for sharing data across sessions. Understanding this layering helps determine where state should live—state strongly tied to a single conversation flow goes in the Checkpointer, while user profiles or facts that need to persist across conversations go in the Store.

How to Choose a Memory Solution

No single memory strategy works for all scenarios. Short customer-service conversations can use Buffer or Window; long-document Q&A tends toward Token budgeting with Summary as a fallback; products that need to remember user preferences across sessions should introduce Long Term and Semantic memory. Real systems often combine multiple strategies: recent messages go through Window, while long-term facts go through Summary or a vector database.

It's worth noting that more memory doesn't equal better results—irrelevant history dilutes the model's attention and actually degrades answer quality. The core of designing a memory system is judging "which information will truly be valuable in future conversations" and letting that information survive at the lowest Token cost.

The original article was recorded by a developer as notes on Dev.to, including some Tamil comments and code screenshots. This article provides a structured reorganization and interpretation based on it.

One-line Summary

Agent memory requires the application layer to store and pass back history; six strategies trade off between cost and information retention.

Paragraph Summary

The LLM itself is stateless—each call requires the application layer to re-pass historical Q&A, and an Agent's memory is essentially storing Query-Response pairs and passing them back as needed. The article outlines six main memory patterns: Buffer stores everything but cost grows linearly with turns; Window keeps only a fixed number of messages at the beginning and end; Token trims by budget but may truncate semantics; Summary compresses old messages into a summary with a cheaper model before deletion; Long Term lets the Agent decide what to store and shares across sessions; Semantic embeds into a vector database and recalls by similarity. Engineering-wise, distinguish Checkpointer (manages a single conversation) from Store (manages cross-session facts), and ensure user_id comes from the application context. There is no one-size-fits-all solution; real systems often combine approaches, and more memory is not always better.