Building a Second Brain for AI Agents: Cohestry with Local Storage and 3D Graphs

Anyone who works with AI Agents every day runs into the same awkward problem: the Agent produces a wealth of valuable context during a session—PRDs, architecture notes, debugging logs, research documents—but once the session ends, all of that becomes unclaimed "dead weight." Next time you start a new session, you have to rely on your own memory to remember which files matter and manually point the Agent to them. A developer on Reddit broke the problem down bluntly: the files exist, but the memory doesn't. So he built Cohestry, a second brain specifically for Agents.

Agents Produce a Lot, but Little Is Reusable

This pain point is not an isolated case. The Agent's working mode makes it a natural "context production machine": it generates design notes while writing code, leaves troubleshooting paths while debugging, and organizes comparison conclusions while doing research. These Markdown files sit in the project directory—in theory they are assets, but in practice they are hard to invoke for the next task.

The author's observation is accurate—what's really missing is not storage, but indexing and recall. People can't remember the ins and outs of an architecture decision made three months ago, and Agents certainly won't proactively dig through old files. The result is that every new session starts from scratch, and the experience accumulated earlier is repeatedly reset to zero. Cohestry's entry point is precisely this gap: turning the scattered notes produced by Agents into long-term memory that can be retrieved and associated.

What Cohestry Does: Local Storage, Automatic Association, 3D Graphs

Cohestry's core design can be broken down into three layers.

The first layer is local storage. All Markdown notes are saved locally, not in the cloud. For people who frequently handle sensitive projects, this point matters more than the features themselves—Agent output often contains internal architecture, key clues, and unpublished research conclusions, and localization is the bottom line.

The second layer is automatic association. The system identifies related notes and establishes connections. The author mentions that the actual value of these connections exceeded his expectations: he can jump along related nodes and rediscover old decisions, old ideas, and project history that he had long forgotten. This is essentially simulating the associative recall of the human brain—not relying on precise keyword search, but on "this note reminds me of another one."

The third layer is the 3D knowledge graph. As usage time grows, knowledge accumulates into a visualized network. The significance of the graph is not that it looks good, but that it makes "what have I actually accumulated" visible. For long-term projects, this global perspective helps uncover buried connections.

MCP Bridge: Letting Agents Find Memories Themselves

What really distinguishes Cohestry from ordinary note-taking tools is the MCP bridge. Once connected, the Agent can actively search this brain using tools and retrieve only the notes it needs, rather than stuffing the entire folder into the context window.

This design solves two practical problems. The first is context bloat: loading all historical documents into the window both wastes tokens and dilutes attention, so on-demand retrieval is the right answer. The second is security: the author mentions that before content is handed to the Agent, secrets such as API Keys and p8 keys are redacted. When an Agent calls external memory, it should not expose credentials along the way.

From an engineering perspective, this is equivalent to equipping the Agent with a read-write protocol for external memory: writing relies on daily output, reading relies on tool calls, and between them sits a layer of retrieval and redaction. Compared with forcing memory into the prompt, this architecture is closer to how humans use notebooks.

MCP (Model Context Protocol) is a protocol proposed and open-sourced by Anthropic at the end of 2024 to standardize the way large model applications connect to external tools and data sources. It is often compared to the "USB-C port for AI applications": in the past, every Agent framework needed custom adapters to connect to databases, file systems, or APIs, whereas MCP defines a unified client-server interaction specification. Tool providers only need to implement an MCP Server once, and any Agent client that supports the protocol can call it directly. In Cohestry's scenario, the memory repository exposes tools such as "search notes" and "read notes" in the form of an MCP Server. The Agent can initiate a tool call when needed, without the platform having to rewrite integration code for each model. This also means the memory layer is decoupled from specific Agent products, so when users switch models or frameworks, the second brain can still be used.

The Second Brain Track Is Heating Up

Cohestry is not an isolated case. Products centered on long-term memory for Agents have been appearing密集ly recently, and their approaches roughly fall into two categories: one is hosted memory services that put memory in the cloud and manage it by the platform; the other, like Cohestry, emphasizes local-first and user control. Each route has trade-offs—cloud solutions are hassle-free but involve data hosting, while local solutions are controllable but require users to maintain them themselves.

The author's approach leans toward the latter and uses "human-readable Markdown" as the underlying format for memory. This is a pragmatic choice: Markdown is both a format that Agents can easily generate and parse and a format that humans can directly browse, edit, and migrate, without being locked into a platform's proprietary format.

What is worth observing is the practical threshold for using such tools. The quality of automatic association, whether the graph remains usable as the number of notes grows, and the recall precision of MCP retrieval all need to be tested in real projects. What the author has provided so far is more personal experience, and data from large-scale use is still lacking.

Conclusion

Cohestry captures a commonly overlooked link: Agents do not lack production capability; what they lack is memory continuity across sessions. Local Markdown storage, automatic association, 3D graphs, and MCP on-demand retrieval—this combination turns the daily by-products of Agents into reusable context assets. For people who run Agent workflows over the long term and also care about data autonomy, this direction is worth watching; whether it can grow from a personal tool into general infrastructure will depend on subsequent performance in retrieval quality and ecosystem integration.

Background Supplement

In engineering, automatic association usually relies on vector retrieval and semantic similarity computation: each note is split and converted into vectors through an embedding model, and then notes with similar content are connected based on cosine similarity. This type of method can capture associations where "the wording is different but the subject is the same" without relying on exact keyword matching. Another common supplementary approach is knowledge graph-style extraction, which identifies entities and relationships from notes to form structured nodes. Both methods have shortcomings—vector retrieval tends to recall content that is semantically similar but topically unrelated, while graph extraction is limited by extraction quality. Therefore, actual products often use a hybrid approach and allow users to manually add or remove connections. When evaluating such tools, the precision of associations deserves more attention than recall, because incorrect connections can lead the Agent to irrelevant context.

Redaction refers to the process of identifying and masking sensitive information before data leaves a controlled environment. A common implementation combines regular expression matching with pattern recognition: detection rules are set for high-entropy strings such as API Keys, private keys, and tokens, and matches are replaced with placeholders. It should be noted that redaction is not foolproof—keys may appear in unconventional formats, or may become sensitive only after being scattered across multiple notes and concatenated. For projects with high security requirements, a safer approach is to store credentials physically isolated from notes, with redaction serving only as an additional line of defense rather than the sole safeguard.