AI Agent Memory Mechanisms: Short-Term vs. Long-Term Memory Explained

Short-Term Memory and Long-Term Memory: Two Core Types

How does an AI Agent remember information across conversations? The answer starts with the model itself: it can only see what's within its context window — all the text it can reference when generating a response. Anthropic's Messages API is stateless, so your application must send the full conversation history with every request. Therefore, any information worth remembering must be stored outside the model and brought back into context when needed — this is Agent memory.

Short-Term Memory and Long-Term Memory: Two Core Types

LangChain categorizes Agent memory into two types. Short-term memory tracks the current conversation — the message history within a single session. There's an example in the OpenAI Agent SDK documentation: the user asks "What city is the Golden Gate Bridge in?" and the answer is "San Francisco"; then the user asks "What state is it in?" and the answer is "California." The Agent can remember this thanks to the session mechanism: before each run, it adds stored history to the input; after each run, it stores new entries.

Long-term memory is kept in a separate store, shared across sessions, and can be recalled at any time in any conversation. For example, the user tells the assistant "Please use Celsius," and the assistant saves it as a memory. A week later in a new session, the user asks about the weather, and the assistant recalls the "Celsius" preference. LangChain calls this type of fact semantic memory. In addition, episodic memory stores experiences, such as past Agent actions; procedural memory stores rules for executing tasks, such as the Agent's system prompt.

Semantic memory, episodic memory, and procedural memory are categories borrowed from cognitive science to distinguish different forms of long-term memory content.

• Semantic memory: stores facts and concepts unrelated to specific experiences, such as "the user's temperature unit preference is Celsius" or "the company headquarters is in Beijing." It answers "what."
• Episodic memory: stores specific experiences tagged with time and context, such as "last Tuesday the user asked me to book a flight to Shanghai." It answers "what happened, when, and where," and typically requires timestamps and event ordering.
• Procedural memory: stores "how-to" skills and rules, such as the response format specified in the system prompt or tool-calling conventions. It's like human muscle memory — once written, it rarely changes.

The storage and retrieval strategies differ for the three: semantic memory suits vector retrieval, episodic memory is often indexed by timeline or session ID, and procedural memory is typically concatenated directly into the system prompt.

Where Is Memory Stored?

Memory is stored outside the model — in files, databases, or vector stores. The video gives two examples:

• Claude's memory tool: Claude can create, read, update, and delete files that persist across sessions on storage controlled by your application.
• LangChain's LangGraph: stores memory as JSON documents (structured text) and can search them semantically via memory tools.

Retrieval: Bringing the Right Memory Back into Context

The process of pulling the right memory back into context is called retrieval, and it works much like RAG (Retrieval-Augmented Generation). Long conversations may not fit entirely into the context window, and memory tools even prompt Claude to "assume interruption" — the context window may be reset at any time. One solution comes from the "correct movement" mentioned in episode three of the video: Anthropic calls it structured note-taking or Agentic memory. Afterward, even if the context is reset, the Agent reads its own notes back into the context window and continues the task.

That's the whiteboard explanation of AI Agent memory mechanisms. To learn how memory fits into an Agent's context, check out the episodes on context engineering.

This article is based on the YouTube video "AI Agent Memory: Short-Term vs. Long-Term, explained on one whiteboard."

RAG (Retrieval-Augmented Generation) is a general paradigm for connecting external knowledge bases to large models: first, documents are chunked and converted into vectors stored in a vector database; when a query arrives, it is also vectorized, the most relevant chunks are recalled by similarity, and these chunks are inserted into the prompt so the model answers based on the retrieved content.

Agent memory retrieval follows the same pipeline, except the "knowledge base" is replaced by the Agent's own accumulated memory entries, and the retrieval signals may be richer — in addition to semantic similarity, you can layer on time decay, importance scores, or session-affiliation filtering. This also means memory systems inherit RAG's typical challenges: inaccurate recall, inappropriate chunk granularity, and similar-but-irrelevant memories interfering with answers.

Structured note-taking means the Agent proactively writes key progress, to-dos, and confirmed facts into a structured external document (such as a headed list or JSON) during a task, rather than relying on the context window to hold all history.

Its difference from ordinary memory retrieval lies in the timing of writes: retrieval is "fetch when needed," while structured note-taking is "write as you go" — a proactive context management strategy. When the context window is reset or compressed due to length limits, the Agent only needs to re-read this note to restore a state where it can continue working, avoiding redundant work or losing intermediate conclusions. Anthropic sees this as a key means of maintaining coherence in long-horizon tasks — essentially treating the context window as a disposable cache and the notes as the persistent source of truth.

One-line Summary

Models are stateless; Agent memory requires external storage and retrieval back into the context window when needed.

Paragraph Summary

Starting from the stateless nature of large models, the article explains why AI Agent memory mechanisms are necessary: the model can only see what's within its context window, and Anthropic's Messages API requires resending the full history with every request, so information worth remembering must be stored outside the model. LangChain divides memory into short-term and long-term: short-term tracks the message history of a single session, while long-term is shared across sessions and further split into semantic, episodic, and procedural memory. Memory can live in files, databases, or vector stores, such as Claude's memory tool and LangGraph's JSON documents. Retrieval works much like RAG, bringing relevant memories back into context; when the context window resets, structured note-taking or Agentic memory lets the Agent read back its own notes and continue the task.