Failed Note Retrieval: Where AI Agent Memory Systems Fall Short — Testing the Retrieval Bottleneck Across 220 Notes

The Notes Are Fine — Not Finding Them Is the Problem

A developer maintains a text note system for a coding agent: one note per topic, with a continuously rewritten current state at the top and an append-only dated history at the bottom. There are currently 220 notes, totaling nearly a million tokens. In each conversation, word-frequency-based search first recommends potentially relevant notes, the agent then requests a map and the cost of each section, and finally retrieves only the one it needs.

The author calls this mechanism a "ladder." In a previous article, he compared this system with Mem0, Zep, claude-mem, and Letta, concluding that "nothing is missing." But that was a "part-for-part" comparison — this equals that. What was missing was a different kind of comparison: given a question, can the line of text that answers it reach the agent's field of view? How many tokens does it cost to bring it there?

This article is that measurement.

Key Findings at a Glance

• Given the correct note, reading the current state and one section finds the answer in 32% of questions at a cost of 2.7k tokens. Loose fragments can't reach this level even with triple the budget.
• Search ranks the correct note first only one third of the time. Under this real-world routing, the ladder narrowly loses to the fragment approach.
• The current state is always findable; the dated history is findable only one fifth of the time. And half of the answers are hidden in the history.
• The previously published 76% hit rate measured something else: whether some useful note was opened, not whether the fact actually arrived.
• 18/18 answers were found when located in the current state; only 21/95 when located in the dated history.

What Memory Benchmarks Actually Measure

LongMemEval, LoCoMo, and the benchmarks published by Mem0 measure the answer, not the document: whether the evidence needed to answer the question reached the model (or whether the answer is correct), and how much context was consumed. Mem0 publishes both token consumption and hit rate — you have to look at both numbers together, because loading everything always hits but is useless.

The author measures the evidence portion: whether the line of text that answers the question entered the content that was read. The previous benchmark measured something else: for each message, which notes were actually opened, and whether search recommended them in the top four, giving 76%. But opening a note doesn't mean finding the answer — a note can have 300 lines. If the agent read the state while the answer sits in some record from last month, the recommendation was correct but the fact never arrived.

Building a Benchmark from Conversation Logs

The author didn't want to hand-write questions, because people write questions their own system is good at answering. The benchmark comes from session logs that Claude Code saves to disk. In each turn, the memory lines read by the agent are collected, and a line counts as an answer if at least three uncommon words in it reappear in the agent's reply and weren't in the user message or the previous three turns.

The second condition is critical: if the words already appeared in the question, the match is an echo, not recall. Turns where the agent wrote to memory are removed, because matches there are freshly drafted content. Note titles are also removed, since search suggestions already include them. The result is 191 matched pairs across 70 notes. Sampling 15 for manual inspection found about 12 were genuine answers — the method is noisy, but the direction is right.

Each line was then searched verbatim in today's notes. Those that no longer exist were discarded — the first unexpected finding: 372 lines that were once answers are no longer written the same way, nearly double the number of surviving ones. Notes get rewritten faster than a benchmark can stabilize. On a system edited daily, hand-crafted memory benchmarks decay within weeks.

BM25 is a classic sparse retrieval algorithm and the default relevance ranking function in systems like Elasticsearch and Lucene. Its core idea: the more a term appears in the current document and the rarer it is across the corpus, the more discriminative it is for that document. Specifically, BM25 is the product of a term-frequency saturation term (TF, with diminishing returns per occurrence) and an inverse document frequency term (IDF, where rarer terms carry more weight), normalized by document length so long documents don't rank highly just for being long.

The so-called "fragment" approach in this article is essentially cutting each note into a pile of small chunks, putting them all into a BM25 index, and querying with the question. Its difference from the "ladder" is not the relevance algorithm itself — both use the same lexical search — but the retrieval unit: fragments retrieve paragraph-level short texts, while the ladder retrieves note-level titles and maps. The finer the retrieval unit, the more concentrated the lexical matching signal, so fragments edge out slightly under real routing; the cost is that the agent receives context-free snippets, lacking structural information like "current state on top."

Four Reading Strategies, Same Budget

Several reading strategies were simulated on these pairs, all single-pass reads of the current notes:

Strategy	Hit rate	Tokens per question
Ladder + correct note given	32%	2.7k
Fragments (best 40)	26%	8.7k
Fragments (best 20)	19%	4.3k
Ladder + real routing (2 notes)	17%	6.1k
Fragments (best 10)	14%	2.2k
Ladder + real routing (1 note)	11%	3.3k
Fragments (best 5)	7%	1.1k
Load everything	100%	977k

Both conclusions are here. Given the correct note, the ladder is the most efficient strategy in the table: 32% hit rate for only 2.7k tokens, while fragments need more than triple the tokens to reach 26%. Grouping knowledge by topic with current state on top works. But under real routing, the same ladder drops below the fragment curve: 11% for 3.3k tokens, versus fragments' 14% for 2.2k tokens.

The note structure isn't the problem. The problem is the step before: choosing which one to open. And I previously measured that step with a ruler that couldn't see the difference.

The author tried two obvious fixes. First, searching within notes for the rarest words in the question — results were 10% and 15%, no change. Second, routing by paragraph and then reading the full note containing the highest-scoring paragraph — results were 7% and 14%. Neither changes the picture, because both rely on the same search.

The 100% hit, 977k token "load everything" baseline deserves separate understanding: it's equivalent to stuffing all 220 notes, nearly a million tokens, into the context window at once, so no matter which line the answer hides in, it necessarily appears before the model. Its role isn't as a usable strategy but as the theoretical ceiling for hit rate and the worst-case reference for token cost. Any efficient retrieval scheme is essentially trading off between "guaranteed hit from 977k tokens" and "almost no tokens but might miss."

Also note how to read "hit rate" and "tokens per question" in the table: they must be compared in pairs, not individually. Sorted by pure hit rate, load everything always wins; sorted by pure tokens, fragments (best 5) is cheapest. The meaningful metric is the minimum tokens needed to reach a given hit rate, which is the premise that makes the article's conclusion — "the ladder is most efficient given the correct note, yet drops below the fragment curve under real routing" — hold.

Why Search Fails Here

Search ranks the note containing the answer first only 32% of the time across these pairs, and within the top four 53% of the time. In the previous benchmark, search across all messages gave 76%. The difference lies in which messages lead to a fact — they're the vaguest ones: "continue with whatever you recommend," "how do we move forward?", "check if everything's fine." These messages give search no usable signal.

The agent still functions because it has the rest of the conversation context and can request another note if the first doesn't work. But every round trip is an extra call, and each call re-reads the entire context. The simulated strategies read once and keep what they read, while the real agent necessarily reaches all these answers by construction — otherwise they wouldn't enter the benchmark. The simulation shows the cost of arriving, not whether arrival is possible.

Routing here doesn't refer to network-layer routing, but to the step of "deciding which notes to open based on a user message." It's the decision step in the memory system pipeline that precedes reading: only if the right note is selected do the downstream reading structures (current state, body, history) have a chance to work. This is the core distinction the article repeatedly emphasizes — note structure determines "how quickly the answer can be found given the correct note," while routing determines "whether that correct note can be obtained."

The two fail through completely different mechanisms. Structural problems can be fixed by rewriting notes and promoting high-frequency answers into the current state; routing problems stem from the limitations of lexical search itself: when the user message is a vague instruction like "continue with whatever you recommend," the question contains almost no usable retrieval signal, and even a strong BM25 has nothing to work with. The article's two "obvious fixes" (searching for the rarest words within notes, routing by paragraph then reading the full note) both fail because they still rely on the same query fed to search — if the query itself has no signal, changing the retrieval granularity just fails in a different place.

History Is the Weakest Area

Given the correct note, whether the ladder finds the answer depends on where the answer sits in the note:

• Current state: 100% (18/18)
• Note body: 28% (22/78)
• Dated history: 22% (21/95)

The current state is always read, so it always hits. History is read by section, and questions almost never mention dates or specific events: they say "that thing last week" or nothing at all. LongMemEval has a category called temporal reasoning, one of the hardest categories for evaluation systems. In this data, half the answers live there.

This changed how the author thinks about notes. The current state works because it's short and rewritten: if something in history gets asked about often, it should be promoted into the state rather than left in a dated entry that must be guessed. History is for auditing why decisions were made, not for fast answers.

Conclusions That Need Correcting

The author once wrote that the system "is already on par with memory projects" and that word-based search "already reaches a 75% hit rate." The first holds by design but is only half true in results: it wins given the correct note, but loses when selecting the note. The second measures whether a useful note was suggested, not whether the fact arrived. Measured with the benchmark's ruler, search ranks the note first only one third of the time.

The author isn't switching systems because of this. Installing Mem0 or a vector database would add fixed text at the start of every session, the most frequently paid part, while fragments' advantage under real routing is only a few percentage points. The lesson is smaller and more useful: before saying something is on par, measure it with the same question others ask themselves.

Limitations

• Lexical vs. lexical: Fragments use BM25 rather than embeddings. Semantic search might improve both the fragment curve and the ladder's routing.
• A hit isn't a correct answer: It measures whether the line of text entered the read content, not whether the model used it well.
• Single-pass reads: The real agent compensates with uncharged calls.
• Exact matching: Lines rewritten in other wording count as lost, and the benchmark discards cases that might be answerable in practice.
• A single person: These are the author's notes, messages, and way of asking, ambiguous when the conversation already has context.

Lexical vs. semantic retrieval is the key to understanding the boundaries of this entire benchmark's conclusions. Lexical retrieval (like BM25) relies on literal co-occurrence matching and can only hit when the question and answer "use similar words"; semantic retrieval (embeddings, vector databases) maps text into high-dimensional space and matches content with similar meaning but different wording via cosine similarity, so it can recall texts like "that thing last week" and "March 14 deployment rollback" that share no literal overlap but are semantically related.

The article deliberately uses only BM25 as a control, a choice that tightens the variables: it ensures the ladder and fragments use the same lexical ruler, thereby isolating the difference of "retrieval unit granularity." But this also means the conclusions can't be directly extrapolated to vector approaches — as the author himself acknowledges in limitations, semantic search could raise both the fragment curve and the ladder's routing hit rate, or even make routing no longer the weakest link. In other words, the finding that "search ranks first correctly only one third of the time" is specific to lexical retrieval, not a universal fate of memory system structures.

FAQ

Is it better to give the agent loose memory fragments or whole notes?

In this data it depends on whether the note is selected correctly. Given the correct note, reading the current state and the most matching section finds the answer in 32% of questions at 2.7k tokens; the best 40 fragments reach 26% at 8.7k tokens. Under real routing — where the note ranks first only one third of the time — fragments edge out slightly under the same budget.

How do you build a memory test benchmark without hand-writing questions?

Build it from conversation logs. Each turn, take the memory lines the agent read, and mark as answers those lines where at least three uncommon words reappear in the agent's reply and weren't in the question. Discard turns where the agent wrote to memory.

Why is the history section of notes the hardest to find?

Because questions almost never mention dates or specific events. Given the correct note, the current state is found 18/18 times, while dated history is found only 21/95. This is what LongMemEval calls temporal reasoning, and half the answers in the data live there.

One-line Summary

Rebuilding a benchmark from conversation logs reveals: note structure is fine, the real bottleneck is that search can't pick out the note containing the answer.

Paragraph Summary

Using real conversation logs to build a benchmark, the article compares two memory reading strategies: the "ladder" (notes grouped by topic, current state on top, history appended by date) and "fragments" (loose text snippets). Given the correct note, the ladder is the most efficient strategy at a 32% hit rate and 2.7k tokens, far exceeding fragments' 26%/8.7k. But under real routing, search ranks the first note correctly only one third of the time, and the ladder loses to fragments. The author further finds that the current state is findable 100% of the time, dated history only 22%, while half the answers hide in history; the previous 76% hit rate measured whether notes were opened, not whether facts arrived. The conclusion is that note structure itself works, the problem is in the earlier step of note selection, and lexical search has almost no signal to work with under vague instructions.