New Standards for LLM Long-Term Memory Evaluation: MERIT Benchmark Reveals the Cost-Performance Tradeoff

Limitations of Traditional Memory Evaluation

Current long-term memory evaluation for AI agents primarily relies on conversational recall benchmarks (such as LoCoMo and LongMemEval), which measure question-answering ability over conversation history rather than whether memorized facts actually change the real-world behavior of tool-using agents. LoCoMo constructs multi-turn long conversation scenarios to test whether models can accurately recall specific details from conversations; LongMemEval focuses on information persistence across sessions. However, both types of benchmarks are essentially "retrieval-QA" paradigms — given historical conversations, the model needs to answer questions about that history. This fundamentally differs from how real AI agents work: in practical scenarios, agents need to translate remembered information into concrete tool calls, API requests, or decision-making actions, rather than merely "stating" what they remember. This evaluation approach therefore has clear shortcomings: it fails to reflect the marginal utility of memory in real task execution and doesn't account for the cost overhead of memory operations.

The term "Tool-Use Agent" refers to AI systems capable of calling external tools (such as search engines, database queries, code executors, calendar management APIs, etc.) to complete tasks. Unlike pure text conversations, tool-use agents need to determine when to call which tool based on context, what parameters to pass, and decide on the next action based on the tool's returned results. Representative frameworks include OpenAI's Function Calling, LangChain's Agent module, and AutoGPT. In such systems, the value of memory lies not just in "what is remembered" but in "whether what's remembered changes behavior" — for example, after remembering a user's preferred delivery address, does the agent automatically fill in that address the next time food is ordered?

To fill this gap, the research team introduced the MERIT (Memory Evaluation for Realistic Instrumented Tasks) benchmark. This framework not only measures memory's actual contribution to task execution but also introduces an explicit cost accounting mechanism, providing a methodology for evaluating LLM agents' long-term memory capabilities that more closely mirrors real-world application scenarios.

Core Design Features of the MERIT Benchmark

MERIT provides scenario-based tool-use tasks across three domains, with the following key design features:

Task Dependency Verification

An automated "leak check" mechanism ensures that task dependencies on facts from earlier events are verified. The leak check design draws inspiration from data leakage detection in machine learning — the mechanism verifies by having the agent execute the same task under "no memory" conditions: if the agent can still complete the task without access to earlier event memories, this indicates the task doesn't actually depend on memory (perhaps because the answer can be inferred from current context, or the LLM's pretraining knowledge already contains relevant information). Only tasks that pass leak check verification — those with a 0 success rate without memory — are included in the final evaluation, ensuring that what's measured is truly the marginal contribution of memory rather than other factors. This means agents must genuinely rely on memory to complete tasks rather than guessing answers through other means.

Difficulty Gradient Design

The test contains task sequences of increasing difficulty, with the ultimate challenge being recall of updated facts. This design enables systematic evaluation of memory system performance across different complexity levels.

Comprehensive Cost Measurement

MERIT performs complete token and dollar measurement for every memory operation. This is the industry's first introduction of explicit cost accounting in memory evaluation, enabling researchers to quantify the return on investment of memory. In LLM applications, tokens are the basic billing unit — both input tokens and output tokens for each API call incur charges. Taking GPT-4.1 as an example, input token pricing is approximately $2 per million tokens, and output tokens approximately $8 per million tokens. Memory operation costs span multiple dimensions: embedding computation or LLM summary generation costs during writes, storage maintenance costs, and retrieval and context injection costs during reads. Full replay — inputting all historical conversations as context — theoretically preserves all information, but as history accumulates, input token counts grow linearly and costs quickly become unsustainable. MERIT is the first to quantify these costs at the dollar level, making "memory ROI" a measurable metric — this has direct engineering guidance value for cost optimization of enterprise-level AI agents.

Experimental Results: Performance Differences Across Memory Systems

The research team invested $42.57 across 23,440 scored episodes in large-scale experiments. Initial pilots used gpt-4.1-mini for two generations of testing, followed by a pre-registered 3-model × 3-random-seed grid experiment (GPT-4.1, Claude Haiku 4.5), with the memory side kept at a fixed configuration. It's worth noting that pre-registered experiments are an important methodological practice for enhancing credibility in scientific research, referring to publicly registering the experimental design, hypotheses, and analysis plan before the experiment begins. In AI research, this practice is still uncommon but is gradually being adopted to address issues like "p-hacking" and "HARKing." The MERIT research team's adoption of this rigorous experimental methodology means they determined the experimental protocol and evaluation criteria before seeing results, setting a new reproducibility benchmark for AI agent evaluation.

Memory Significantly Improves Task Success Rate

In dependency tasks, memory improved success rates from a leak-verified 0.00 baseline to 0.55–1.00. This confirms that for tasks genuinely dependent on historical information, long-term memory has irreplaceable value.

The Performance Gap Between Memory Implementations

Experiments revealed the enormous impact of memory implementation on performance:

Instability of Embedding Retrieval: In updated-fact tasks, embedding retrieval methods exhibited unpredictable collapses, with success rates fluctuating between 0.30–0.95, with a maximum gap of 0.45 across different random seeds. Embedding Retrieval is currently the most mainstream long-term memory implementation and the core technology of RAG (Retrieval-Augmented Generation) systems. Its workflow involves first converting historical conversations or events into high-dimensional vectors via embedding models (such as OpenAI's text-embedding-3-small, Cohere Embed, etc.), storing them in vector databases (such as Pinecone, Weaviate, ChromaDB); when recall is needed, the current query is similarly converted to a vector, and the most relevant historical segments are found through cosine similarity or approximate nearest neighbor search. The advantage of this approach is that no predefined storage structure is needed, but its vulnerability lies in the fact that semantic similarity search can fail in fact-update scenarios — when the same entity has both old and new values, retrieval may return outdated information, and retrieval result relevance is highly dependent on embedding model quality and query phrasing. Even more concerning, even when the correct value was retrieved, agents only acted on it 55% of the time, exposing a huge gap between "retrieving information" and "acting on information."

Stable Advantage of Structured Storage: In contrast, write-time update storage systems (including structured fact stores and LLM summarization) maintained stable performance at 0.70–1.00. Write-time update storage is a proactive memory management strategy with two main implementations: structured fact storage organizes information as key-value pairs or knowledge graphs (e.g., "user preferred address: XX Road, Chaoyang District, Beijing"), directly overwriting old values when new information arrives, avoiding the confusion of old and new information coexisting; LLM summarization leverages the large language model itself to compress and integrate historical memories — whenever a new event occurs, the LLM merges new information with existing summaries to generate updated summaries. The common characteristic of both methods is that information integration is completed at write time rather than handling conflicts at read time, which explains why they perform more robustly in fact-update scenarios. Interestingly, LLM summarization performs excellently as a memory mechanism, but at the cost of additional LLM inference overhead for each write.

Unexpected Disadvantage of Hybrid Approaches: The study found that hybrid approaches actually performed worse than using fact storage alone, challenging the intuitive assumption that "more is always better." This phenomenon likely stems from signal conflicts between multiple memory sources — when embedding retrieval returns outdated information while structured storage holds the latest value, agents may make incorrect decisions due to contradictory information, performing worse than relying on a single reliable source.

Spot checks on the latest generation of models (Claude Sonnet 5, based on clean full-replay controls) reproduced this pattern, confirming the robustness of the findings.

Cost-Benefit Analysis: Full Replay Is Not the Optimal Choice

A key finding of the research is: full replay is never cost-effective. Full replay refers to feeding all historical interactions between the agent and user verbatim as context input to the LLM. While this approach offers the highest information fidelity, token consumption per call grows dramatically as interaction history increases. Switching memory implementations can change task success rates by up to 60 percentage points, and the optimal configuration for each domain delivers 2.7–3.9× dollar returns on its marginal utility.

This finding has important implications for practical deployment: in production environments, choosing the right memory architecture is more critical than simply increasing memory capacity. Cost awareness should be a core consideration in memory system design. For enterprise applications handling millions of daily interactions, the choice of memory architecture directly impacts operational costs — a poor choice could mean tens of thousands of dollars in additional monthly spending without corresponding performance gains.

Research Significance and Future Directions

The release of the MERIT benchmark marks a new phase in LLM agent memory evaluation. By incorporating task execution, cost accounting, and multiple memory implementations into a unified framework, researchers can now more accurately assess the true value of memory systems.

The research team has open-sourced the complete benchmark, testing tools, and all trace data, providing valuable research infrastructure for the community. Future research directions may include: exploring more efficient memory compression techniques (such as information-theory-based optimal compression strategies), designing adaptive memory strategies (dynamically switching memory implementations based on task type and frequency), and validating the generalizability of these findings across more domains. Additionally, with the emergence of long context window models (such as Claude and Gemini supporting 200K token contexts), the tradeoff between memory systems and long-context capabilities will also become a direction worth deeper investigation.

For developers, this research sends a clear signal: when building tool-using AI agents, memory system selection and configuration should be carefully weighed based on specific task characteristics and cost constraints, rather than blindly pursuing larger memory capacity.

Key Takeaways