Building RecallAI in Four Days: A Graph Database Memory System Retrospective

Why AI Needs a "Memory System"

Have you ever experienced this: you explained your learning difficulties to an AI chatbot a few days ago, and when you come back days later, you have to start all over again? This is exactly the problem Aman Kumar and his friends wanted to solve. They teamed up to participate in the Graph Hacks hackathon jointly organized by WeMakeDevs and FalkorDB, chose the "Agent Memory and Coordination" track, and spent four days building an AI tutor prototype called RecallAI, attempting to make AI remember users' learning history and provide personalized answers.

The team consisted of four people: Aman Kumar, Rishu, Abhishek, and Ayan, with members from different backgrounds and varying levels of proficiency in LLMs, graph databases, and front-end/back-end integration. Their goal was not to build a perfect product, but rather, within limited time and hardware constraints, to figure out one core question: how exactly does persistent memory make AI applications more useful?

The value of this experience lies in its authenticity—many hackathon projects only showcase the successful side, while the RecallAI team honestly documented the confusion over architecture choices, the failure of the memory system, and the decision-making process of having to cut features on the fourth day to preserve the core experience.

Architecture Design: Four Components and One Key Choice

On the first day, the team clarified the basic architecture of RecallAI, which includes four main parts:

• Frontend: The chat interface where users interact with the AI
• Backend: Handles requests and coordinates the memory retrieval process
• Language Model: Understands questions and generates answers
• FalkorDB: A graph database used to store and retrieve relationships between information

They chose FalkorDB hoping to explore graph-based memory structures rather than treating each conversation as an isolated text block. Behind this choice was a reasonable intuition: in the AI tutor scenario, there is a natural relationship network among students, knowledge points, and learning records—for example, a "difficulty" relationship between "a certain student" and the knowledge point "dictionary," and a "participation" relationship with "a certain learning session." Modeling these relationships with graphs theoretically makes retrieval more targeted.

But a clear architecture does not equal smooth implementation. The team quickly discovered that getting the four components to communicate correctly was far more difficult than expected.

"Knowing clearly what you want to build is one thing; figuring out how the parts work together is another."

Day Two: LLM Context Windows Are Not Permanent Memory

On the second day, the team delved into how language models work with memory systems. An important realization they gained was that language models do not automatically acquire reliable permanent memory just because you continue a conversation. The information a model can use depends on the context you provide, as well as the capabilities of the model itself and the application layer.

Here is an easily confused concept: fine-tuning and persistent memory solve different problems. Fine-tuning changes model behavior through training samples, while persistent memory allows the application to externally store information and retrieve it when needed. For a hackathon project with tight time and limited hardware, training a model from scratch was unrealistic, so the team chose Qwen2.5-1.5B-Instruct—a relatively small instruction-tuned model.

This choice was pragmatic. The team only had one reasonably performing laptop that could run local workloads, with some work relying on cloud resources. Rather than spending time wrestling with large models, it was better to focus energy on the memory architecture. But small models also bring an easily underestimated challenge: they still need appropriate context, and the application still needs reliable retrieval and supply mechanisms.

"A database can store information, but that doesn't mean the AI automatically knows when and how to use it."

Context Window is the maximum text length (usually measured in tokens) that a language model can process at one time. It's like the model's "workbench": the conversation history you put into this request, the retrieved materials, and the system prompt must all fit into this window. Content exceeding the window is truncated, and the model simply cannot see it. Therefore, the context window is a "temporarily visible range," not a "permanent memory store"—as conversation turns increase, earlier content is gradually pushed out or compressed. The value of a persistent memory system lies in: storing important information externally (such as in a vector database or graph database), and only re-injecting the most relevant fragments into the window when needed, thereby breaking through window capacity limits and preserving information across sessions. Note that window size and model parameter scale are two independent dimensions: a small model can also have a large window, and a large model's window can also be limited. When designing memory applications, you must consider both "what to store" and "when to retrieve and stuff it into the window."

Fine-tuning is continuing to train a pre-trained model with samples from a specific task or domain, thereby changing the model's weights and behavioral tendencies. It's suitable for teaching the model "how to say things"—such as tone, format, and domain terminology. But it is not suitable as "memory": retraining every time new user data is added is extremely costly, and the fine-tuned model makes it difficult to precisely delete a piece of information or guarantee isolation by user. Persistent Memory, on the other hand, stores information outside the model (databases, files, vector indexes), retrieves it on demand during inference, and stitches it into the context, with model weights unchanged. The benefits are that information can be added, deleted, modified, and queried, can be audited, can be isolated across users, and updates require no retraining. The two are often used together: fine-tuning makes the model better understand "how to use retrieved memories," while external memory is responsible for "remembering specific facts." In the hackathon scenario, the team abandoning fine-tuning in favor of external memory was a reasonable trade-off under the dual constraints of resources and maintainability.

Day Three: Why the Memory System Failed

The third day was the most critical turning point of the project. The team connected the components and tested them, and at first it seemed very promising—the system could sometimes recognize keywords or recall information when directly referenced. But when questions depended on broader conversational context, the results were unstable.

Sometimes the AI behaved as if it had no memory; sometimes it gave an answer that sounded confident but did not actually reflect previous conversations. The team realized that remembering a specific phrase and truly understanding its surrounding context are two different things. The system might successfully retrieve a keyword but fail to connect that information to the user's current question.

The problem could occur at any link in the chain:

1. Was the information actually saved correctly?
2. Was the correct information retrieved from FalkorDB?
3. Did the backend pass the retrieved information to the language model?
4. Did the model receive enough context to understand the association?
5. Did the model ignore or misinterpret the information it received?

These five links form a complete processing chain, and a problem in any one of them makes the entire memory system appear to fail. The team was unable to confirm and resolve all causes one by one during the hackathon, but the troubleshooting process itself made them understand one thing: a successful database connection absolutely does not mean the AI application has reliable memory.

"Remembering a keyword is not the same as remembering a conversation. Retrieving information is not the same as understanding it."

Retrieval-Augmented Generation (RAG) is a general paradigm for making models "look up information before answering": convert the user's question into a query, recall relevant fragments from an external knowledge base, stitch them into the prompt, and then have the model generate an answer. The reliability of RAG depends on the entire chain: whether chunking and indexing are reasonable, whether the query is accurate, whether recall is relevant, whether ranking puts the most useful content first, and whether the prompt clearly requires the model to use this content. The "sometimes remembers, sometimes doesn't" that the team encountered in the article is a typical manifestation of instability in one link of the RAG chain. If using a graph database, retrieval can be "relationship traversal" rather than pure vector similarity: first locate the entity (such as the knowledge point "dictionary"), then follow edges to find associated learning sessions and difficulty records. The advantage of graph retrieval is that it can carry relationship semantics; the disadvantage is that it depends on the accuracy of entity recognition and graph schema design. Regardless of the method, retrieved content must be explicitly placed into the context, and the model must be clearly told "this is available historical information," otherwise the model may ignore it.

Day Four: Cut Features, Keep the Core

By the fourth day, there was a natural urge to add features—study plans, auto-generated practice questions, progress tracking, even multi-agent collaboration. All these ideas could make RecallAI a better learning companion.

But the team made a counterintuitive decision: no new features. If the system cannot reliably remember and use previous conversations, piling more features on top will only make the application more complex. They focused their efforts on six core behaviors:

1. Save useful information: Store relevant details from conversations rather than relying entirely on the current chat
2. Retrieve the right memories: Find relevant information even when the user changes their wording
3. Pass information to the model: Provide retrieved context when generating answers
4. Use context correctly: Ensure answers reflect relevant information rather than generating irrelevant content
5. Avoid irrelevant memories: Don't dredge up irrelevant old information in new questions
6. Isolate user memories: Ensure information stored by one user does not appear in another user's conversation

These six were not all achieved perfectly within four days, but clearly identifying the gap between goals and reality is itself important progress.

"When core functionality is unreliable, piling on more features doesn't help."

End-to-End Evaluation refers to testing the complete chain from user input to final answer, rather than only verifying whether individual components report errors. For memory systems, several types of probe test cases can be designed: ask the same fact in three different phrasings to see if all can be recalled; discuss topic A then topic B to see if old information from A is incorrectly brought in; two users alternate questions to see if memories cross over; ask about a new topic never stored before to see if the system fabricates old memories. For each type of test case, check the intermediate artifacts: what retrieval returned, what the prompt passed to the model contains, and whether the model's answer references this content. Testing only "the database can connect" and "the model can respond" is not enough, because information can be silently lost between components. The value of end-to-end evaluation is in turning "looks like it works" into "reproducibly reliable," and establishing a regression baseline for subsequent iterations.

Key Lesson: AI Applications Are a System Problem, Not a Component Problem

Looking back at the entire project, the team believes the biggest takeaway was not the specific usage of any framework, model, or database, but understanding that AI applications must be considered as complete systems.

Before starting, it's easy to think that connecting a language model to a database is enough to have memory. The reality is that the real challenge lies in coordinating the entire process: storing information, finding the right information later, passing it to the model, and ensuring the model uses it appropriately. Each link needs to be verified individually and also end-to-end.

Hardware limitations actually helped the team make more focused decisions. Choosing Qwen2.5-1.5B-Instruct was not because it was the "best" model, but because it allowed the team to spend time on the memory architecture. But the small model also proved one thing: a small model does not mean a simple application. The model still needs appropriate context, and the application still needs reliable retrieval and supply mechanisms.

The most valuable realization is perhaps this: a feature is not complete just because the components are connected; it needs to work reliably from the user's perspective. The database is connected, the model can generate answers, the frontend looks complete—but if information cannot flow correctly between them, the experience will still fail.

Next Steps for RecallAI

The team's long-term goal is to make RecallAI a more personalized AI learning companion: remembering which concepts you get stuck on, identifying whether you repeatedly make the same mistakes, and adjusting explanations based on your learning history.

But to get there, the basic memory system must first be made more reliable. The team's next steps include: testing memory storage and retrieval more systematically, understanding why broader context is sometimes lost, verifying whether retrieved information is actually passed to the language model and used, and testing the system's behavior when there are no relevant memories.

Real memory is not just storing information. It means connecting the right information to the right question at the right time. This is the problem the RecallAI team is continuing to tackle, and it is an unavoidable challenge for any developer who wants to build AI applications with memory capabilities.