Tencent Hy Memory Tested: The Write Model Determines Memory Success

Title

Tencent Hy Memory Tested: The Write Model Determines Memory Success

Meta Description

Testing Tencent Hy Memory 2.0 on the LongMemEval benchmark: with a cheap write model it scores only 43%, while Claude Opus writing matches the no-LLM approach but costs up to $5 per user. A deep dive into the core trade-off between write quality and cost in memory systems.

Article Content

Tencent open-sourced Hy Memory 2.0 this week — an Apache-2.0 licensed framework, a 30B memory model, and MemHub (an app that shares a single memory across multiple AI tools). As the author of Aura, an open-source project in the same space, I care about how it performs on the two most practical failure scenarios:

1. The fact has been updated, but it still answers with the old version;
2. Forgetting "who said what", especially things the assistant itself said.

Disclosure of interest: I am the author of Aura, an open-source memory library with a Windows app. All test configurations and passing thresholds were fixed before running, and unfavorable results are reported honestly.

Test Methodology

I used the public LongMemEval benchmark. Each question comes with roughly 50 turns of user chat history, with the answer hidden inside. I selected three of the six question categories, 20 questions each:

Category	What it tests
Knowledge Update	A fact changed later — does memory give the new version?
Assistant Said	The answer exists only in what the assistant said
User Said	The answer is in the user's words (control group)

Both systems received identical histories and questions. Both used the same model to read the retrieved memory and answer (Gemini 3.8 Flash), and the same model scored them using the official LongMemEval judge prompt.

• Hy Memory 2.0.1, default pro mode. Hy uses an LLM to convert each conversation into facts and reconcile them with old facts. Tencent's own 30B model has no available API, and the framework supports any OpenAI-compatible model (its README uses DeepSeek^Ad). So I ran it twice: once with Gemini 3.8 Flash and once with Claude Opus 5.5 as the write model. Opus is a fair substitute — Tencent's model card compares its model with Claude Opus 5, with nearly identical average scores (55.5 vs. 54.6).
• Aura 1.62.0 preserves each turn of conversation as-is, with dates and speakers, and retrieves the 10 most relevant items. No LLM is used at write time; embeddings run locally.

LongMemEval is a public benchmark for evaluating long-term memory systems. Its core design: it constructs chat history for a single user spanning multiple sessions (here about 50 conversations), then appends questions that require cross-session reasoning to answer. Questions are divided into several capability dimensions, such as knowledge update, temporal reasoning, multi-session reasoning, assistant memory, and user memory, testing whether a system can locate and integrate information from lengthy history. Judging is typically done by another LLM using the official prompt to semantically compare the system's answer against the gold answer, rather than by string matching. Because the history length, question types, and judging method are all publicly fixed, LongMemEval is often used to compare the accuracy of different memory approaches horizontally — but this also means results are influenced by how the history is constructed and which judge model is chosen.

Write model (write-time LLM) refers to the model a memory system calls during the save stage, independent of the reading model used to answer questions. In this article, for example, Hy uses an LLM at write time to convert each conversation into structured facts; the stronger the write model, the higher the quality of fact extraction and old-new reconciliation. Aura does not call an LLM at write time and only preserves the raw conversation. This distinction matters because writing happens during the history accumulation stage, with cost growing linearly with the amount of user data, whereas reading only happens at question time. When evaluating a memory system, if you only report final accuracy without stating the write model's strength and number of calls, it is hard to judge whether the approach's cost is acceptable at real scale.

Result 1: With a Cheap Write Model, Hy Answers Only 43% Correctly

Question Type	Aura	Hy (Gemini Flash write)
Knowledge Update	90%	55%
Assistant Said	100%	0%
User Said (control)	95%	75%
All 60 questions	95%	43%

Of Hy's 34 failures, 32 were "I don't have that information."

The first rebuttal I anticipated: Hy's 10 facts are shorter than Aura's 10 conversation turns (about 2,000 vs. 5,000 characters), so maybe it simply reads less content. I re-searched Hy's storage with a larger limit, without rewriting anything. When the text volume matched Aura's, it scored 45%; retrieving 40 memories (about 7,900 characters), it still scored only 45%. The information was lost when conversations were converted into facts, not at the retrieval stage.

Result 2: With Claude Opus Writing, Hy Matches on Fact Updates

Using Opus as the write model costs about $5 per question, so I only ran 10 questions per category.

Knowledge Update, same 10 questions:

	Correct
Aura (no LLM at write time)	9 / 10
Hy, Gemini Flash write	6 / 10
Hy, Claude Opus 5.5 write	9 / 10

This round went against me. With a strong write model, Hy's ability to track changing facts is comparable to Aura's. It stores old values as history ("current: 25:50 … history: 27:12"), which is the correct design. Both missed the same question: two different counts appeared on the same day.

Assistant Said, 8 questions (Anthropic quota ran out on 2 of 10, excluded):

	Correct
Aura	8 / 8
Hy, Gemini Flash write	0 / 8
Hy, Claude Opus 5.5 write	4 / 8

Opus helped a lot here: it does store some of the assistant's suggestions as memory ("assistant suggested…"), which Flash never did. But half of what the assistant said still failed to make it into memory. All four failures were "I don't have that information," even after retrieving 40 memories.

Cost Comparison at Equal Accuracy

For a user history of about 50 conversations:

	LLM calls at write time	Tokens	Write time
Hy, Gemini Flash write	84	416K input, 9K output	about 15 minutes
Hy, Claude Opus 5.5 write	92	695K input, 61K output (about $5 at list price)	about 11 minutes
Aura	0	0	36 seconds

This is where the real trade-off lies. Hy's quality is bought by having an LLM read conversations turn by turn at write time. With a cheap model, its score on changing facts drops to 55–60%; with Opus, it can match a memory approach that never calls a model — at a cost of about $5 per user history.

Root Cause: Rewriting Conversations into Facts Loses Information

Hy compresses each conversation into statements about the user ("the user is interested in learning and cooking classic Peruvian dishes"). This makes memory small and tidy, but discards everything the write model deems not a user fact: the assistant's recommendations, seemingly minor details, and sometimes the updated version of a fact. Here's what happened when running with Flash:

• "Have I finished reading The Nightingale?" Hy said the book was set aside for being too emotional; later the user had finished it.
• An earlier goal of 100 was overwritten by a new goal of 150, so "what was my previous goal?" had no answer.
• "Which beer did you recommend for that recipe?" The recommendation came from the assistant, so it wasn't stored.

Aura takes the opposite route: it preserves what was said verbatim, along with speaker and time, leaving the decision of what matters to the model answering the question. This requires more storage and gives the reader more text, but requires no guessing at write time.

Fact reconciliation is a key step in Hy-like systems: after a new conversation is extracted into facts, the system must decide whether it adds to, duplicates, or replaces existing facts. If a new fact conflicts with an old one (e.g., a goal changed from 100 to 150), the correct approach is to demote the old value to history and mark the new one as current, thereby supporting both "what is it now" and "what was it before" questions. Reconciliation depends on the write model's judgment; a cheap model easily overwrites old values outright or ignores conflicts, leading to "the fact was updated but it still answers with the old version." And if conflicting values are all kept without marking timeliness, retrieval may recall the wrong version. Aura does not reconcile in advance; it preserves the raw words along with timestamps, letting the reader model decide which version is newer when answering.

Limitations

• Only three of LongMemEval's six categories were tested, chosen because they correspond to the two failure scenarios above. Multi-session reasoning, temporal reasoning, and preferences were not tested — in those scenarios, summarizing conversations into facts might actually help.
• Small sample size. The main run had 60 questions; the Opus runs had 10 and 8 questions. Each question corresponds to a swing of 5–12 percentage points.
• Not Tencent's own model. Its 30B model might write better than Flash. Opus is the closest comparison provided by its model card.
• Only pro mode was tested. ultra (graph mode) was not tested.
• A single reading and judging model (Gemini 3.8 Flash), the same for both systems.
• In the main run, 1.3% of Hy's conversations failed to be stored (empty text was sent and rejected by the model), affecting at most 1–2 questions.

Conclusion

If you're evaluating a memory system that uses an LLM for summarization, check two things on your own data: how strong a write model it needs to perform well, and how much that model costs per user. Also ask whether it preserves what the assistant said — because "what did you recommend last time?" is a question users really ask.

Aura is open source (MIT): github.com/teolex2020/aura-memory. The Windows app shares one local memory across tools like Claude Code, Cursor, Codex, and Gemini CLI: aurasdk.dev/desktop.

One-line Summary

Tencent Hy Memory 2.0 uses LLM summarization to write memory, scoring only 43% with a cheap model and losing what the assistant said.

Paragraph Summary

Writing as the author of Aura, the writer compares Hy Memory 2.0 and Aura on three LongMemEval categories. Hy compresses conversations into user facts and uses an LLM to reconcile them; Aura preserves the raw conversation, speaker, and timestamps with no LLM at write time. With cheap Gemini Flash writing, Hy answers only 43% correctly, and 32 of its 34 failures are "I don't have that information." Switching to Claude Opus writing, it matches Aura on Knowledge Update (9/10 vs. 9/10), but Assistant Said is still only 4/8. The root cause is that fact-based writing discards assistant recommendations, minor details, and old version information, and expanding retrieval volume cannot compensate. In terms of cost, Hy needs about $5 and nearly a million tokens per user history, while Aura uses zero LLM calls and takes 36 seconds.