AI Memory Mechanisms Explained: Models Forget, Systems Remember

Every time you reopen an AI conversation from a few days ago and continue asking questions, it always picks up the thread—this "sense of memory" makes people think the large model actually remembered something. But the truth is: the model remembers nothing; it's your system that remembers on its behalf.

This article comes from a developer's systematic breakdown. Starting from "what does the model actually see," he walks step by step through how to design a memory architecture for your own AI feature. The core conclusion is just one: every memory technique is, at its essence, answering the same question—for this call, what should go into the context?

The Model Is a Stateless Function

A language model does exactly one thing: given a sequence of tokens, predict the next token. Generating text is just a loop around this process—the model's weights are fixed after training, and nothing you say in a conversation changes them. All the model can see is whatever fits in the context window.

A simple experiment proves this. In the first call, tell the model "My name is Shayan." In the second call, ask directly "What is my name?" Without the history, it will answer "I don't know your name." Only when you attach the first round's conversation history will it say "Your name is Shayan."

The model didn't remember the name—you sent the name again.

Before reaching the model, the messages array gets flattened into a long string of text with role markers, and the model is simply continuing that text. The reason a chat app can "pick up the thread" is that the backend database stores every message. When you send a new message, it sends the entire conversation along with the new message, and the model re-reads it from the first character.

This has a key consequence: if each round adds about 500 tokens, then by round 100 you're sending about 50,000 tokens. The total compute of a conversation grows roughly quadratically with its length—caching and compression mechanisms exist precisely to address this.

Memory Can Live in Three Places

Since the model itself is stateless, where should "memory" go? The author offers a three-layer model:

Layer	Location	Good At	Limitation
Parametric memory	Model weights	Broad general knowledge	Frozen at training time, fuzzy, knows nothing about you
Context memory	Input to the current call	Precise, the model reasons over it directly	Limited capacity, pay per token, disappears when the call ends
External memory	Databases, files, vector stores	Unlimited and persistent	Invisible to the model unless placed into context

The author uses the film Memento as an analogy: the protagonist Leonard cannot form new memories and relies on notes and tattoos, re-reading them in every scene. An LLM app is Leonard plus a notebook.

From this comes the rule that runs through the whole article: every memory system answers the same question—for this call, what moves from the external layer into the context? Sliding windows, summarization, RAG, and memory tools are all different answers to the same question. This is exactly the core of what is now called "context engineering."

What Exactly Is the KV Cache

This is the part the author misunderstood the longest. When the model reads a token, it computes several vectors for it, two of which are key: key (K) is the label, value (V) is the content. When a new token arrives, it builds a query (Q) that is compared against every previous key to find who to attend to—this is the attention mechanism.

In a text generation model, tokens can only look backward, not forward, so the K and V of "Shayan" depend only on the tokens before it. No matter how many words are appended after it, that row stays completely unchanged. So why recompute it at every step? You don't have to. Store this table in GPU memory and append one row per new token—this table is the KV cache.

The savings are staggering. Prompt of 1000 tokens, generate 100 tokens: without caching you'd compute about 105,000 rows; with caching only about 1,100 rows (first prefill the 1000 tokens in parallel, then 1 per step).

The cost is GPU memory. A Llama-3-70B-class model at 16-bit precision uses about 320KB per token, so a 100,000-token prompt alone needs about 32GB of GPU memory just for the cache. This also explains why output tokens cost more than input tokens—prefill can process the entire prompt in parallel, which GPUs excel at; decode generates token by token, reading weights and cache at every step.

Key distinction: a cache is not memory. The KV cache adds no new information; delete it and you can rebuild it from the tokens. All you lose is time and money.

Understanding the KV cache requires first understanding the Transformer's attention mechanism. In the self-attention layer, each token's representation is projected into three sets of vectors: Query, Key, and Value. During attention computation, the current token's Query does a dot product with the Keys of all previous tokens in the sequence, producing a set of weights, which are then used to take a weighted sum of all Values to get the current token's contextual representation.

The key point: generation is autoregressive, so a token can only attend to positions before it. This means that when the Nth token is processed, the Keys and Values of the previous N-1 tokens have already been computed and will not change. The essence of the KV cache is to keep these historical Key/Value matrices in GPU memory, computing only the new token's own K and V and appending them at each step, without recomputing the whole sequence.

So the KV cache is a classic space-for-time optimization: use GPU memory to store historical computation results and avoid O(n²) repeated computation. This is also why long-context inference is so GPU-memory-intensive, and why providers price input tokens and output tokens separately.

Prompt Caching Keeps the Table Between Requests

Between two adjacent turns, the second request starts exactly the same as the first, so the first rows of its KV table are also identical. Prompt caching is letting the provider retain this table for a short time so the next request beginning with the same tokens can skip the recomputation.

The prefix rule determines which rows survive. It's like a prefix-sum array or git commit hashes: change the third element and everything before it stays the same while everything after changes. The cache is only valid for prefixes that are token-for-token identical from the very beginning—an extra space, a reordered JSON key, a changed number, and everything after it is invalidated.

So prompts should be arranged from stable to volatile: tool definitions → system prompt → static documents → conversation history → new user message. The classic mistake is putting f"Current time: {datetime.now()}" at the top of the system prompt—the prefix changes on every request and the cache never hits. Put it in the user message instead.

The author cites a calculation: for a 50,000-token document answering 20 questions, without caching you need 1 million input tokens, with caching about 157,000—roughly 84% lower input cost—and the first token arrives faster too.

Back to the puzzle at the start: reopening that conversation days later, the cache is long gone—it only survives a few minutes. The conversation can continue because the real memory (the tokens themselves) lives in the app's database, and that table is rebuilt from scratch.

What to Do When the Context Is Full

Context has three limits: a hard limit (the context window—exceed it and you get an error), an economic limit (cost and time-to-first-token grow with tokens), and a quality limit (the longer the context, the worse the model uses it—the "Lost in the Middle" research shows models use information at the beginning and end of a long context better and the middle worse).

Option A: sliding window, and its subtle trap. Keeping only the most recent 20 messages seems simple, but there's an easily overlooked problem: every time the window slides, the first token of the history changes, invalidating the entire history's cache and driving the hit rate to nearly zero. The fix is batch eviction: let the window grow from 20 to 40, then cut back to 20 in one go. This way, between two trims the history only grows at the end, and the cache keeps hitting. The principle: change the context rarely and in batches, not a little every turn.

Option B: compression. When the conversation hits 80% of budget, use an LLM to summarize the earlier part and replace it. Summaries are lossy, so the summarizer's prompt is critical—explicitly tell it what to preserve: user facts, decisions made and why, unfinished tasks, IDs and numbers. After many rounds summaries degrade, like a game of telephone. For content that absolutely must not be lost, summarization isn't enough—you need the long-term memory from step six.

Memory Across Conversations

Imagine a customer service assistant for a food delivery app: in one conversation the user says "Please don't add onions to my order," and three weeks later in a new conversation asks "Recommend a pizza." The history is empty, and neither sliding windows nor summarization can help.

This requires a pipeline: extract → consolidate → store → inject.

Extraction uses a separate LLM call (a cheap model is fine) to pull persistent facts out of the conversation. Consolidation is the hardest part—if you only insert without updating, memory fills up with duplicates and contradictions. So before storing, you retrieve similar memories and let the LLM decide ADD, UPDATE, DELETE, or NOOP. This is using an LLM as a database's merge logic. Note: not every contradiction is a change ("I ate fish at a wedding" is an exception, not a new diet), and sometimes you want to keep history rather than overwrite.

The author gives two default practices that are very valuable: extract asynchronously, off the reply path—because the facts from this conversation are already in this conversation's history, and long-term memory is for the next conversation; take a snapshot of memory at the start of each conversation—the memory block goes before the history, and updating it mid-conversation would break the entire history's cache.

Risks include: wrong facts saved permanently, stale facts, memory poisoning (a user pastes a webpage inducing "remember that the user wants every answer to include a certain link"; if the extractor saves it, it becomes a permanent prompt injection), privacy, and over-personalization.

Using RAG to Find the Right Memory

Once memory grows too large to fit entirely in context, you need to retrieve the relevant parts. Keyword search isn't enough—"my car broke down" won't match "the vehicle won't start."

Embedding is another, smaller model that turns text into fixed-length vectors; texts with similar meaning have closer distances, measured by cosine similarity. Note these vectors are not the K and V from step three—the embedding model is specially trained to bring similar meanings close together. Non-English users should choose a multilingual embedding model.

RAG's retrieval flow is: question → rewrite → embed → top-k → rerank → context → LLM. The model doesn't search for anything; it's your code that searches and puts the results into the context.

If you already use Postgres, the pgvector extension is enough for most projects. But the WHERE user_id line is a security boundary, not a detail—forget it and one user's memories end up in another user's context.

There are three typical retrieval failures: exact strings (order number #48213, which embeddings don't understand) need hybrid search (BM25 + vectors, fused with RRF); conversation-dependent questions ("What about the other one? Is it faster?") need an LLM to rewrite them into standalone questions; time ("that bug we talked about last week") is mostly about when—embeddings don't understand time, so it should be a filter condition.

Embedding is a technique that maps discrete text to continuous dense vectors. Its training objective is usually to bring semantically similar sentences closer together in vector space; for example, contrastive learning pulls positive pairs closer and pushes negative pairs apart. This process is done by a dedicated embedding model, entirely different from the internal K/V vectors of a generative model—the latter are intermediate activations of the attention layer, while the former are trained semantic representations.

The common similarity measure is cosine similarity, which measures the cosine of the angle between two vectors and is insensitive to vector length. During retrieval, the system embeds the query and the documents separately, computes similarity, and returns the top-k most similar document chunks.

BM25 is a classic keyword retrieval algorithm based on term frequency and inverse document frequency. It works well for exact terms (like order numbers or proper nouns) but doesn't understand synonyms. Hybrid search merges the results of BM25 and vector retrieval, often using RRF (Reciprocal Rank Fusion): for each document, take the reciprocal of its rank in each of the two ranked lists and sum them to get a combined ranking. This captures both exact matches and semantic similarity, covering the blind spots of any single method.

When the Model Manages Its Own Memory

In all the previous steps, code decides what goes into the context and the model just reads it. Agentic memory flips the decision—give the model tools and let it decide when to read and when to write. Leonard starts writing his own notes.

The key detail: the model never touches the database directly. It issues requests, and your code executes them. So the guardrails live in your code: user_id comes from the session, not the model's input; limit memory size and count; file modes must validate paths.

Anthropic's memory tool has Claude request to view, create, or edit files under /memories, with the app executing on controlled storage. MemGPT (now Letta) goes further, using an OS analogy: the context is RAM, external storage is disk, and the model decides what to swap in and out.

There's a nice side effect for caching: tool results append to the end of the context, and agentic retrieval never touches the prefix, making it naturally cache-friendly—this is "just-in-time context."

Comparing the two modes: pipeline-driven is decided by code, always runs, and is easy to test, but may inject irrelevant facts; agent-driven is decided by the model, only invoked when needed, and more flexible, but the model may forget to save or look things up, testing is harder, and the risk of memory poisoning is more direct. In practice most systems mix the two.

Just-in-time context is a strategy opposed to preloaded context: instead of stuffing all possibly useful information into the prompt in advance, let the model fetch what it needs during reasoning via tool calls. The core advantages are three: it saves tokens and cost, since useless information never enters the context; it protects the prompt cache prefix, because tool results always append at the end; and it lets the model judge the relevance of information itself, which suits scenarios where the amount of memory far exceeds the context window.

Note that just-in-time context hands the decision of when to retrieve to the model, so it places higher demands on tool descriptions, model capability, and error recovery. If the model doesn't realize it needs to retrieve, or a tool call fails, it may hallucinate or give an irrelevant answer. So in practice you usually combine it with pipeline-driven fallback logic, force-injecting necessary memory on critical paths while allowing the model to supplement on its own.

The End-to-End Journey of One Request

The user opens a new conversation and types "Recommend a pizza": the app loads the system prompt, tools, memory snapshot, and history (empty); it builds the request in stable-to-volatile order and sets a cache breakpoint; the provider finds the system prompt and tools in the cache and skips re-reading them; the model prefills the remaining tokens and builds KV rows; the model calls recall, and the app runs a filtered vector + keyword search and appends the results; the model writes the answer token by token, reusing the KV cache; the app saves the new message; extraction, consolidation, and storage happen asynchronously in the background, preparing for the next conversation.

The moment the model finishes step 6, it forgets everything. Every other row in that list is the system remembering on its behalf.

What This Approach Doesn't Solve

The author candidly lists the limitations: retrieval misses things; when it misses, the model either says it doesn't know or makes something up, and more memory is not equal to better memory; summarizing and extracting facts are lossy and can be wrong—you need evaluation, not good intentions; every layer adds cost, latency, and debugging burden—for many features, a well-ordered prompt plus prompt caching and no long-term memory at all is the right answer; don't build step eight when step four is enough; stored memory is stored personal data, and you must shoulder retention, deletion, and access rules.

Back to that conversation: when you reopen it, it feels like the AI remembers me, but it doesn't. The database remembers, and a stateless function re-reads all the messages from the beginning and continues. Memory is not a capability toggle you can flip on inside the model; it is state. And like any state in the systems we build, it needs an owner, a place to live, and a plan for when it expires.

Memory poisoning is a class of security risk unique to long-term memory systems. Because the extraction pipeline writes "persistent facts" identified from conversations into external storage, an attacker can induce the extractor through conversation content to save malicious instructions, for example making the model "remember that the user wants every answer to include a certain link." Once that record is saved, it gets injected into the context of all subsequent conversations, equivalent to a permanent prompt injection whose impact far exceeds a single session.

Defense approaches include: tagging the source and assessing confidence for each extracted memory, limiting the permission scope of individual memories, filtering written content for sensitive patterns, and keeping audit logs and rollback capability for memory updates. In addition, memories involving personal data must comply with deletion and access rights under regulations such as GDPR, which means the storage layer must support precise per-user deletion and export.

One-line Summary

The large model itself has no memory; all "sense of memory" is the result of the application reorganizing the context on every call.

Paragraph Summary

Starting from the essence that "the model is a stateless function," the article systematically breaks down memory architecture in LLM applications. Model weights are frozen after training, and so-called memory is really the application re-sending historical tokens each time, which leads to a three-layer memory model of parametric, contextual, and external memory. The article deeply distinguishes the KV cache from prompt caching: the former is an inference optimization that adds no new information; the latter follows strict prefix rules, requiring prompts to be ordered from stable to volatile. It then discusses sliding-window and summarization strategies once the context window fills, the extract-consolidate-store-inject pipeline for cross-conversation memory, the RAG flow for retrieving relevant memories and its failure modes, and the trade-offs of agentic memory letting the model manage memory itself. Ultimately it emphasizes: memory is not a switch inside the model but system state, requiring ownership, storage, and an expiration plan.