Ditch the Vector Database: Building a Memory Layer for LangChain Agents with BM25

The Agent Amnesia Problem

Developers who have built agents with LangChain have likely run into the same frustrating problem: the agent does something useful in one session, then forgets everything by the next. The industry's standard fix is to bolt on a vector database — configure an embedding model, chunk your text, and hope that cosine similarity returns the right context.

This approach works, but it comes at a cost. You need a persistent service, an embedding model, and you take on real latency and expense. One open-source developer shared an alternative on Reddit: a memory layer called CogniCore that provides persistent memory for agents using BM25 retrieval and a multi-hop knowledge graph — no vector database, embedding server, or cloud dependencies required.

BM25 Instead of Embeddings: Lightweight Without Sacrificing Effectiveness

The most fundamental design decision in CogniCore is replacing embedding vectors with the classic BM25 retrieval algorithm. The author emphasizes that the retrieval engine is implemented entirely using Python's standard library — after a pip install, agents can store memories, retrieve them, and reflect on past experiences, with no heavy infrastructure overhead.

For LangChain users, the most practical aspect is that it works as a memory backend alongside existing pipelines. You don't need to replace your agent — you add a memory layer underneath it. The agent stores what it learns after each step, then retrieves those memories before acting in the next session.

Benchmarks: Beating Embeddings in Small-Context Scenarios

The author ran comparisons on the LongMemEval benchmark, with results that are somewhat counterintuitive. At small context sizes, BM25 actually outperforms embedding-based retrieval:

• At 5 chunks: accuracy improves from 78.8% to 85.2%
• At 10 chunks: from 87.2% to 92.8%
• At 20 chunks: both converge at around 95%

The author's explanation is that brute-force retrieval "closes the gap" by the 20-chunk mark, at which point embeddings gradually catch up. This data is a useful reminder for developers: in scenarios with limited context size, simple keyword-frequency retrieval may not fall short of expensive semantic embeddings — and it wins on latency and cost.

An Unexpected Bonus: Agent Memory Migration

The most surprising part of the project for its author was an "agent memory migration" system he built along the way. One agent can export its validated memories — memories backed by "evidence": what commands were run, what the results were, what the exit codes were — and another agent can import them.

The receiving agent gets a packaged memory bundle with a content hash for tamper detection and a full chain of custody indicating the source of each memory. The author reports testing this across Figma, Groq, ElevenLabs, and Claude, moving memories between entirely different platforms and modalities.

A Trust Model Inspired by Reinforcement Learning

The trust mechanism for this migration system loosely borrows concepts from reinforcement learning:

• Validated memories function like an experience replay buffer — only actions proven to work get stored.
• When memories flow between untrusted agents, their confidence scores decay, analogous to a discount factor in RL.
• Receiving agents can set policies to outright reject memories that fall below a confidence threshold or lack verification.

This design turns "memory trustworthiness" into a quantifiable, transferable, and rejectable asset — offering a novel approach for multi-agent collaboration.

A Brief Assessment

The author openly acknowledges being the creator of the project, which is open-sourced on GitHub (cognicore-dev/cognicore-env) and published to PyPI (cognicore-env). He says his intent isn't to pitch a product, but to share what he learned about agent memory through the building process.

From an engineering perspective, CogniCore's value lies in challenging the default assumption that agent memory must rely on a vector database. For scenarios with limited context size and a need for low latency and zero external dependencies, a BM25-based approach is genuinely worth exploring. That said, the benchmark data comes from a single self-reported source, and the convergence at 20 chunks shows that embedding-based solutions still hold advantages at larger retrieval scales — this approach is better suited as a lightweight complement than a wholesale replacement. The memory migration trust model is a more experimental frontier, and its reliability in production environments still needs broader validation.