LLMs Are Stateless Functions: The Nature and Pitfalls of AI Memory

Every "memory" in an AI product is, at its core, someone somewhere deciding what to put into an array. This insight comes from a series of technical articles about building a code-map agent, where the author uses a pitfall hit by a forty-line program to explain LLM memory thoroughly.

A Bug That Looks Correct

After the author finished the forty-line version of the code-map tool, they asked it to read a second file and answer "what's the relationship between this file and the previous one." The model shot back: which file are you talking about?

The first instinct was that some flag got dropped or a session id wasn't passed. So they spent far more time than expected hunting for this non-existent session id. The conclusion was brutal: the endpoint had no idea it had been called 90 seconds ago, and no configuration could change that.

The root of the problem was hidden in a piece of code that looked completely correct. The function signature takes a message list, returns a reply — it reads like a conversation. But messages + [...] just builds a temporary list, sends it, and throws it away. The caller's list was never modified. Every call started from the same starting point.

This code passes code review because the type annotations imply the existence of "history" — you have to notice that the list never grows.

There Is No Session, Only a List

The fix takes just three lines, and the core idea is: the caller owns the list.

The first line appends the user message, the middle calls the model, and the third appends the assistant's reply. The function deliberately mutates the list passed in — there's only one conversation record, the caller holds it, and every append happens inside a function where you can set a breakpoint.

The second append is the easiest to miss, and its failure mode is worse than having no memory at all. If you don't append the assistant's reply, the model receives a string of questions with no answers — a conversation record in which it seemingly never spoke. So it re-answers questions it already answered, overturns conclusions from two turns ago, and behaves like a model with very poor reasoning. You'll waste an entire evening rewriting system prompts and initial instructions before you find the problem.

Putting both appends in the same function isn't a code-style issue — it eliminates the state of "only recording half the conversation."

Carrying the full array on every request means the first turn gets billed again on the second turn, and billed a third time on the third turn.

Stateless is a fundamental property of LLM API calls: every HTTP request is independent, the server doesn't retain any cross-request context for you, and it doesn't automatically splice the previous turn's user message and assistant reply into the current request. The input the model receives during inference is a complete message array, and the output is simply a pure computation result based on that array — after computation, the server retains nothing. This means so-called "multi-turn conversation" is entirely fabricated by the client repeatedly sending a growing array. Understanding this is crucial: many beginners assume the endpoint "remembers" what was just said, and waste time looking for a non-existent session id or implicit session mechanism. The fact is, as long as the caller doesn't include the history in the request, the model is physically incapable of seeing it.

Your Agent Has Only One State

Because the model is a pure function of the array, the array is the program's complete state. So write it to disk.

One line of code ends the main function, and transcript.json goes into .gitignore. When the output goes wrong, the question you actually need to answer isn't "why did the model say that" but "what did the model see." Only one artifact can answer that question, and now it's on disk.

Almost every bad answer you chase down at this stage can be traced in the file within thirty seconds: some file was never appended, source code was read as an empty string because the path was relative to the wrong directory, a system message was overwritten by a subsequent assignment. None of these are model problems, but before you open the conversation record, they all look like model problems.

This also makes runs reproducible. The input is a JSON array, and you can resend it byte for byte, change one message, and send it again.

The Bill Is a Triangle, Not a Line

Carrying the entire array on every call — if each turn adds roughly T tokens, the nth request carries about nT input tokens, and the cost of the entire run is T·n(n+1)/2. Ten turns equals the input volume of fifty-five turns; twenty turns equals two hundred and ten.

For a chat assistant this is tolerable, because one turn is one sentence. But for an agent reading source files, each turn is an entire file, T is enormous, and the triangle becomes terrifying within single-digit turn counts.

Three honest strategies, each with trade-offs:

• Send everything: cost grows quadratically with turns; suitable for short sessions and learning phases
• Summarize old turns: cost plateaus after a threshold, but loses details you didn't know were important at the time; suitable for chat scenarios
• Keep pointers and re-read on demand: cost plateaus plus one call per lookup; suitable when content is on disk and re-reading is cheap

For a codebase, the third is obviously correct — the repository is still there, and re-reading a file is just a disk read. This is also why agents ultimately need to fetch content themselves rather than being fed content.

Token is the basic unit of text processed by an LLM. An English word is usually split into one to a few tokens, and a Chinese character corresponds to roughly one to two tokens. Billing counts input tokens and output tokens separately, and this section is mainly about the input side. The so-called "triangle" growth comes from two superimposed effects: first, every turn resends all previous turns' messages; second, as the number of turns increases, the history carried in a single request grows longer. Multiply the two and you get the cumulative formula T·n(n+1)/2. This also explains a common confusion: why does an agent session that looks like only ten turns produce a bill that feels like dozens of turns? In the scenario where an agent reads source files, T itself can be as high as thousands or even tens of thousands of tokens, and the triangle quickly goes from "acceptable" to "unbearable."

Practical Advice

1. Pass the conversation record as a parameter. Once it becomes a hidden property on an object, you can no longer see what you sent.
2. Append both sides' messages in the same function. Make the "only record half" state impossible to express.
3. Write the array to disk on every run, starting from day one. Four lines of code, and it's your only observability tool for a long time.
4. Assert role ordering in tests. Checking that two turns of conversation produce the sequence system, user, assistant, user, assistant will immediately catch the bug above.
5. Don't install a memory library before you can clearly articulate which strategy you need. All three options are a page of code; the hard part is choosing, and no library can make that decision for you.

"Memory" is a product word. Every implementation of it is a strategy about what to put into the array. Choosing that strategy well is the core of most of the work that follows.

One-line Summary

LLMs are stateless functions, and AI "memory" is nothing more than a strategy the caller chooses for what to put into the message array.

Paragraph Summary

Drawing on a pitfall hit while building a forty-line code-map agent, the article thoroughly explains the nature of AI memory: LLMs are stateless functions, each call only sees the message array carried in the request, and the endpoint remembers nothing about prior interactions. The author uses a bug that looks correct to illustrate this — the code constructs a temporary message list, sends it, and throws it away, so the caller's list never grows. The fix takes just three lines, and the core is to let the caller own and mutate this conversation record, and you must append both the user message and the assistant reply in the same function, otherwise the model receives a string of questions with no answers and behaves like it has very poor reasoning. Carrying the full array on every request makes input cost grow as a triangle of T·n(n+1)/2. The article offers three strategies — send everything, summarize old turns, keep pointers and re-read on demand — and emphasizes that writing the conversation record to disk is the foremost observability tool.