Agent Memory System Design: Short-Term Memory, Long-Term Memory, and Context Management

When building an AI Agent, the memory system is often the most underrated component — yet it has the greatest impact on product experience. It was never about "whether the model can remember," but rather "which information, at what time, and in what form enters the context window." In a technical sharing session, Bilibili creator "码士集团" systematically broke down the design methodology of Agent memory systems. This article combines their core insights with practical engineering experience for a complete overview.

Why Agents Can't Do Without a Memory System

An Agent without a memory system can essentially only handle single-turn, short-chain tasks. Once a task requires cross-step execution, or the user expects it to remember long-term preferences, problems surface immediately.

The creator gave a very typical example: a user explicitly requested the day before that "code should be concise, variable names short, minimal comments." The next day, when asked to write a feature, the Agent still outputs lengthy Chinese comments. The reason isn't that the model deliberately ignores the request — it's that the previous conversation simply doesn't exist for the current session. There's no way for it to remember those specific rules.

More serious problems arise in multi-step tasks. If the Agent doesn't know what was looked up in the previous step, where it currently stands, or what information has already been collected, subsequent steps can't connect, and the final output can be disastrous.

This is precisely why the memory system is the watershed that takes an Agent from a "Q&A tool" to a "persistent assistant."

The Division of Labor Between Short-Term and Long-Term Memory

Agent memory typically falls into two categories: short-term memory and long-term memory, each playing a distinctly different role.

Short-Term Memory: The Workbench Inside the Context Window

Short-term memory corresponds to the message sequence in the model's context window, holding the complete running state of the current task. It includes SystemMessage, UserMessage, AIMessage, and ToolMessage — user inputs, model outputs, and tool call results are all appended to the message sequence.

Short-term memory has three notable characteristics:

• Easily lost: Once the session closes, all context information in the window is gone;
• Limited capacity: Whether the context window is 200K or 1M tokens, it's only a matter of time before it fills up;
• Directly readable: The model can respond based on prior content when processing subsequent conversation.

Think of it as the Agent's "workbench" — information for the task currently being handled is spread out on it, but the workbench has limited space. Once full, you must find a way to handle it, such as compression, summarization, or a sliding window.

Long-Term Memory: A Persistent Repository Across Sessions

Long-term memory stores persistent information that spans tasks and sessions. It typically resides on external devices, with common carriers including vector databases, relational databases, and cache databases. Information isn't lost after a task ends and can be retrieved when needed next time.

The most critical capability of long-term memory is semantic retrieval: it doesn't depend on exact keywords — anything semantically similar can be recalled. This comes with a clear prerequisite — if you're using a vector database like Milvus, semantic matching works; but if you store data in MySQL or Redis, semantic recall isn't directly possible.

Basic Principles of Semantic Retrieval and Embedding: Semantic retrieval means the system doesn't rely on specific keywords in the user's query, but instead recalls content by comparing the similarity of "semantic vectors." The key to this is the Embedding model — it maps a piece of text into a high-dimensional floating-point vector (e.g., 1536 or 1024 dimensions), where semantically similar texts are closer together in vector space. During retrieval, the system encodes the user's query into a vector as well, then uses metrics like cosine similarity or inner product to find the closest records in the vector store. This is why the original text emphasizes that "storing in MySQL or Redis doesn't allow direct semantic recall": those two databases excel at exact matching, range queries, and key-value reads, and inherently lack vector similarity computation capabilities. If you really want to use them for semantic retrieval, you typically need to introduce additional vector indexing plugins (such as Redis's RediSearch or MySQL's vector extensions); otherwise, you can only fall back to keyword matching.

The Relationship Between Context Window and Tokens: Many people interpret "200K context" as being able to hold 200,000 Chinese characters, which is inaccurate. The unit of the context window is tokens, not characters. For Chinese, 1 Chinese character typically corresponds to 1–2 tokens; for English, about 4 characters correspond to 1 token. Therefore, a 200K token window can actually hold roughly 100,000 to 200,000 Chinese characters, depending on the tokenizer and text type. More importantly, tool call return results and code snippets often have higher token density and consume the window faster. Understanding this is essential for making reasonable estimates of when "sliding window" and "summary compression" should trigger and at what compression ratio.

Three Core Questions in Designing a Memory System

The creator emphasized that designing a memory system isn't about saving all historical information — it's about managing the information lifecycle. The core is answering three questions: what to store, how to store it, and when to retrieve it.

What to Store: Use "Does It Make the Agent Perform Better?" as the Criterion

The judgment criterion can be direct: if the Agent knew this piece of information at the start of the next task, would it perform better? If yes, it's worth storing; if not, storing it serves no purpose other than occupying the context window.

Information worth persisting typically falls into three categories:

1. User preferences and habits: language style, tech stack preferences, working style;
2. Key conclusions and decisions within a task: rule-based content like "competitor pricing uses pay-as-you-go";
3. External knowledge: product documentation, Q&A notes, standard workflows, historical case archives, etc.

What's not worth writing to long-term memory includes: intermediate reasoning processes, raw logs returned by tools, and small talk. This information dilutes the memory store and reduces retrieval accuracy.

How to Store: Match Different Information to Different Storage Methods

Not all data should go into a vector database. The reasonable approach is to route by information type:

• Unstructured text requiring semantic retrieval: suitable for vector databases, recalled via embedding;
• Structured preference and state fields: suitable for relational or cache databases, with precise and efficient queries;
• Entire knowledge bases or documents: can combine vector databases with RAG.

In practical engineering, hybrid storage is common — structured fields go through precise queries, unstructured knowledge goes through vector retrieval, and the two are used in conjunction. Technology selection is never singular; flexible combination based on business scenarios is the norm.

When to Retrieve: Active Retrieval and Passive Triggering

There are two strategies for using memory:

• Active retrieval: Before a task begins, use the current task description to query the memory system, inject the results into the System Prompt, and let the Agent enter the task with background context;
• Passive triggering: Encapsulate memory queries as a tool, letting the Agent decide autonomously during reasoning when to call it.

The best results come from combining both strategies: load user preferences at the start of a session, and retrieve specialized knowledge and historical data on demand during execution.

What to Do When Short-Term Memory Exceeds the Context Window

After multiple rounds of complex tasks, tool returns and historical messages quickly fill the window. Without any management, new information can't get in, old information gets truncated, and the Agent experiences "amnesia." There are three common approaches:

Sliding window: Keep only the most recent few rounds of conversation. Simple to implement, but may lose key constraints from earlier conversations.

Summary compression: As the limit approaches, use an LLM to compress early history into a summary, replacing the original messages. For example, a 200K history might compress down to just 10K. This significantly reduces window usage but loses detail — the summary only retains general content.

External offloading: Write intermediate results that aren't currently needed but may be useful later into long-term memory, remove them from context, and retrieve them back when needed. This is essentially adding an external storage extension to the context window.

These strategies aren't complex to implement in engineering. If you need more systematic encapsulation, you can look into open-source components specifically providing memory capabilities for Agents, such as memory and letta, which offer more complete abstractions of these strategies.

What Is RAG: RAG (Retrieval-Augmented Generation) refers to retrieving content relevant to the current question from an external knowledge base before the large model generates an answer, then feeding that content as context into the model. It addresses the problem of private knowledge that isn't included in the model's parameters or has become outdated. RAG and long-term memory are highly similar in mechanism — both are "retrieve first, then inject into context." The difference is that RAG typically targets static domain knowledge documents, while long-term memory leans more toward dynamic, personalized information that accumulates as the user interacts. In practical systems, the two often share the same vector database and retrieval pipeline, differing only in write sources and update frequency. Understanding RAG helps developers reuse existing retrieval infrastructure when designing memory and avoid reinventing the wheel.

The Complete Memory Processing Pipeline

A complete memory module typically operates in three stages:

Before the task begins — read memory: Read user preferences, project configurations, and historical decisions from structured memory. Simultaneously use the task description to perform semantic retrieval against long-term memory, retrieve relevant background, and inject it into the System Prompt. The Agent works from the very first step with constraints and background.

During task execution — use memory: Short-term memory works throughout. User inputs, model outputs, and tool call results are continuously appended to the message sequence. Every call to the large model carries this history, so the Agent always knows the current progress, completed tasks, and pending items.

After the task ends — write memory: Write new information generated by this task back to persistent storage, update fields in structured memory, and write valuable conclusions into long-term memory after embedding. Short-term memory is simply cleared — released when the session closes.

The value of this pipeline is that the Agent can continuously accumulate across multiple uses, understanding the user better and the domain knowledge better over time.

The Essence of a Memory System

The value of a memory system isn't in remembering everything — it's in getting the right information into context at the right time, giving the Agent continuity, consistency, and long-term adaptability.

This also explains an easily overlooked point: the memory system must be continuously updated. Many people think that once a storage structure is designed and a batch of data is loaded in, the job is done. But if you want the Agent to get stronger over time, the memory store needs continuous iteration. The creator mentioned that when tools like Open Cloud were popular early on, users were encouraged to write a Memory.md file to record user preferences, usage habits, and behavioral guidelines. This way, every time a new session starts, the Agent automatically follows your rules without you having to restate requirements repeatedly.

For Agents targeting specific business scenarios, industry background and domain knowledge in long-term memory are even more essential. Only by accumulating this content can the Agent's feedback truly match the needs of a specific industry and business.

Treat the memory system as a core component to design, not an afterthought patch — this is a lesson every Agent developer should invest effort in.

One-line Summary

An Agent memory system coordinates short-term and long-term memory to get the right information into context at the right time.

Paragraph Summary

The article systematically outlines the design methodology for AI Agent memory systems. The core idea is: memory isn't about whether the model can remember, but which information, at what time, and in what form enters the context window. Memory is divided into short-term (message sequences in the context window — easily lost, limited capacity) and long-term (persistent information in vector, relational, and cache databases, supporting semantic retrieval). Design requires answering three questions: what to store — using "does it make the Agent perform better" as the criterion, keeping only user preferences, key conclusions, and external knowledge; how to store — routing structured and unstructured information separately, often using hybrid storage; when to retrieve — combining active retrieval with passive triggering. When short-term memory exceeds the window, three strategies apply: sliding window, summary compression, and external offloading. The complete pipeline has three stages: reading before the task, using during execution, and writing after completion. The memory store must be continuously updated so the Agent understands the user and domain better over time.