Building an AI Agent from Scratch: File-Based Long-Term Memory Implementation and Security Design

In this tutorial series on building AI Agents, we've already equipped the Agent with capabilities like tool calling, long-task planning, human intervention, and security policies. But one fundamental flaw has always persisted: once the program shuts down, the Agent forgets everything. It might have just learned that you prefer concise answers, or that the project uses uv to manage Python commands, yet on the next startup it has to rediscover everything from scratch.

This article is the latest installment in the series, focusing on adding long-term memory capabilities to the Agent. The original author uses a file system rather than a database or vector store, implementing persistence in a minimalist way, and discusses the security boundaries around memory.

Limitations of Short-Term Memory and the Role of Long-Term Memory

The messages list in the Agent loop is itself a form of memory—it contains the system prompt, user messages, model replies, and tool results. This is short-term memory, which lives with the process and is sent to the model as context. But it has two hard limits: it disappears when the process ends, and it cannot grow indefinitely because the model's context window is finite—every additional token costs time and money.

Long-term memory is different: it survives across sessions and is only retrieved when needed. The original article divides it into two categories: project memory (such as environment facts and project conventions, e.g., "Python commands use uv") and user memory (stable preferences, e.g., "prefers concise answers"). This scoping gives each type of memory a clear purpose and boundary.

Context window refers to the maximum number of tokens a model can receive in a single inference (including input and output). Window sizes vary widely across models, from a few thousand to over a million tokens. Content exceeding the window is truncated or causes the request to fail, so the Agent cannot indefinitely stuff all historical messages into every request. This is the fundamental reason short-term memory cannot replace long-term memory: even if the process doesn't restart, history will exceed the window limit as the task progresses. Common coping strategies include sliding windows (keeping only the most recent N turns), summarization/compression (condensing old messages into short text), and on-demand retrieval (fetching only memory fragments relevant to the current task). The file-based long-term memory in the original article is precisely a minimalist implementation of the on-demand retrieval approach—historical notes are stored on disk and only enter the context when the Agent actively calls the read tool.

Privacy Issues and Security Rules Brought by Memory

Memory makes the Agent more useful, but it also introduces new privacy and security issues: memory is persistent data, and a bad or overly broad note may affect subsequent tasks long after the original context has disappeared.

The original article sets strict rules in the system prompt: read project memory and user memory at the start of a session; only append concise, useful facts that are likely to remain true in future sessions; never store secrets, credentials, or temporary task details. Specific security considerations include: API keys, passwords, and tokens must never enter the model context or persistent files; one-off instructions (e.g., "use blue headings for this document") are not worth long-term storage; prompt injection content from web pages or files must never be promoted to memory; notes should be concise enough for humans to review and correct.

By default, the memory directory is located outside the Docker project mount, and the Agent can only access it through MemoryStore, not its file system tools. This architecturally isolates memory data from project files and reduces the injection attack surface.

Prompt injection refers to an attacker hiding malicious instructions in content the Agent will read (such as web pages, documents, or tool return values) to induce the model to perform unintended actions. When the Agent has memory write capabilities, the harm of injection is further amplified: an attacker could induce the model to write malicious instructions into persistent memory files, making them persist across all subsequent sessions and creating cross-session persistent contamination. The original article explicitly states that "prompt injection content from web pages or files must never be promoted to memory" precisely to cut off this attack chain. Another related concept is the data minimization principle: collect and store only the minimum information necessary to complete the task, avoiding long-term retention of sensitive data. The original article's requirement for concise notes and exclusion of secrets and temporary details is essentially practicing this principle at the memory level while reducing the cost of human review and correction.

Implementation of File-Based Memory Storage

The original article chooses not to introduce a database or vector store—a few files are enough—easy to inspect, easy to back up. The default directory is ~/.agent-harness/memory, containing two append-only note files: project.md and user.md.

The core MemoryStore class provides two methods: read_notes reads notes for a specified scope (returns "(empty)" if the file doesn't exist), and append_notes appends content (writes a newline first if the file is non-empty). The internal _notes_path helper function only accepts the values project and user, and throws ValueError for any other input—the model cannot choose arbitrary file names, which is a simple but effective form of path control.

The Agent accesses memory through two tools: read_memory (reads notes for a specified scope) and append_memory (appends concise persistent notes). Like the scratchpad and todo list, they run in-process because they operate on harness state rather than project files. These two tools are categorized under PLANNING_TOOLS, so each call does not trigger a permission confirmation—a reasonable default given that they only access the configured memory directory, but it also means the memory directory should be treated as sensitive user data and protected with OS permissions and backups.

Allowlist is an access control strategy: only pre-defined, explicitly listed options are permitted, and all other input is rejected. Compared to a blocklist (which lists prohibited items), an allowlist denies by default and is harder to bypass because attackers cannot break through by constructing unexpected input. The original article's _notes_path only accepts the two scope values project and user, throwing ValueError for any other input—this is the allowlist approach. It matters because the parameters of memory tools are generated by the model—if the model were allowed to specify arbitrary file names, an attacker could use prompt injection to induce the model to read or write files outside the memory directory, causing path traversal or leakage of sensitive files. Hardcoding the selectable values in code rather than relying on the model to voluntarily follow prompt constraints is an embodiment of defense in depth: prompts can be bypassed, but validation in code cannot.

Practical Test: Cross-Session Memory in Action

The original article provides test records from two sessions. In the first session, the user asks: "I prefer concise answers, please remember this for future sessions." The Agent first reads user memory (returns empty), then appends "The user prefers concise answers." and confirms it will keep answers concise in the future.

In the second session, the user asks: "Help me understand this project." The Agent reads project memory, getting "Use uv for Python commands"; reads user memory, getting "The user prefers concise answers"; then responds: "I'll inspect the project and keep explanations concise." This shows that memory persists correctly across sessions and is actively retrieved at the start of a new task.

Interestingly, the Agent does not indiscriminately load all history but instead reads the two scoped notes on demand through tools, avoiding stuffing the full history into every model request.

What's Done and What's Next

The current implementation provides two key capabilities: scoped notes stored as human-readable Markdown files for stable environment facts and user preferences; and memory security rules that limit saved content to concise, persistent facts, explicitly excluding secrets and temporary data.

The original article teases the next step: building session management on top of this—listing historical sessions, resuming unfinished sessions, and compressing overly long sessions so they can continue to run within the model's context window. For developers building their own Agents, this path of starting with files and gradually layering security constraints offers a more controllable evolution path than jumping straight to a vector database.

The code can be cloned from the series' GitHub repository (basic-agent-harness).

One-line Summary

Implement cross-session long-term memory for an AI Agent using the file system, with security rules constraining stored content.

Paragraph Summary

This article explains how to add long-term memory to an AI Agent. The author points out that the messages list in the Agent loop is only short-term memory, limited by both process lifecycle and context window, and cannot retain information across sessions. Long-term memory is divided into project memory (environment facts and conventions) and user memory (stable preferences), stored as Markdown files in a separate directory. On the security side, the article emphasizes never storing secrets, credentials, or temporary task details, prohibiting promotion of prompt injection content to memory, and restricting file access scope through the MemoryStore class and an allowlist mechanism. Tests show that memory persists correctly across sessions and is retrieved on demand, laying the foundation for future session management features.