Tencent Hy-Memory-30B-A3B Local Deployment in Practice: How the Memory Writer Crushes Opus 5

Title

Tencent Hy-Memory-30B-A3B Local Deployment in Practice: How the Memory Writer Crushes Opus 5

Meta Description

Complete guide to local deployment of Tencent Hy-Memory-30B-A3B: a 30B MoE memory writer, Q4_K_M only 18.2GB, runs on a 24GB GPU or 32GB Mac. Includes llama.cpp, Ollama, vLLM, and MLX deployment commands, plus an in-depth look at benchmarks—the so-called "surpassing Opus 5" is only 0.87 points, but the 11-point gain over Qwen3-30B-A3B is the real highlight.

Article Content

Tencent has open-sourced a model with "memory" in its name—Hy-Memory-30B-A3B. Interestingly, it doesn't store any memory itself; it's a memory writer: a 30B-parameter mixture-of-experts (MoE) model that activates about 3B parameters per token, specifically trained to filter which information should be written to an external memory store and how to update old entries. The actual memory data still lives in the vector database of your Mem0, MemOS, or Hy-Memory backend.

The core value of this model lies in replacing the expensive process of calling GPT/Claude on every write in memory frameworks. According to the model card, Hy-Memory-30B-A3B was trained in two stages: memory on-policy distillation and "Attributed" memory reinforcement learning, all completed within real memory frameworks. It's released under the Apache-2.0 license, so commercial use requires no additional legal communication.

Specs and Hardware Requirements

Judging from the model card and config.json, Hy-Memory-30B-A3B uses the HYV3ForCausalLM architecture, with 128 routed experts, 8 activated per token, 48 layers, hidden dimension 2048, max position embeddings 262,144, and support for Chinese and English.

In actual deployment, the cost of the A3B architecture needs to be understood: only about 3B parameters participate in computation, so decoding is fast, but routing may select any of the 128 experts, so all 30B weights must reside in memory—you pay for memory at 30B, save on compute at 3B.

Sizes by version: Q8_0 GGUF is 32.0 GB, Q6_K 24.7 GB, Q5_K_M 21.4 GB, Q4_K_M 18.2 GB; MLX mixed 4/6-bit is about 17 GB; original BF16 is about 60 GB. Adding KV cache and system overhead, the realistic threshold is a 24 GB GPU or a 32 GB Mac running Q4_K_M; to go Q8_0 with long context you'll need 64 GB. Note that the 262K max context is a configuration value—the GGUF uploader measured only up to 32K in practice.

Mixture-of-Experts (MoE) and what the A3B name means: MoE splits the feed-forward layer into multiple "expert" subnetworks, and a router selects Top-K experts to participate in computation for each token. In Tencent Hy-Memory-30B-A3B, "30B" refers to the total parameter count of all experts, and "A3B" means about 3B parameters are activated per token (8 of 128 routed experts). This explains the line "you pay for memory at 30B, save on compute at 3B": VRAM/RAM must hold all expert weights because any token could route to any expert; but only about 3B actually participate in matrix multiplication, so decoding throughput is close to that of a 3B-class model. The costs of MoE also include routing imbalance and the fact that different experts being quantized to different precisions can amplify errors—which is why the community sees "Q4_K_M runs but strict success rate drops."

Benchmarks: The Truth Behind the 0.87-Point Edge

Tencent's published comparison table swaps only the memory writer while keeping the answer model, judge, and retrieval fixed. The six-benchmark average shows Hy-Memory-30B-A3B at 55.49, slightly above Claude Opus 5's 54.62—a 0.87-point gap. But looking at individual items: 0.02 points under the Hy-Memory framework, 0.01 under MemOS—essentially noise level. These are numbers the vendor ran on a 20% subsample of each benchmark using its own evaluation framework. OrcaRouter's comment hits the nail on the head: this model "deserves to be tested, but not to be blindly trusted."

The truly valuable part is the lower half of the table: compared with Qwen3-30B-A3B, often used as a cheap local extractor, the average gain is 11 points; on the EverMemOS framework never seen in training, it still scores 56.89. If you're using Mem0 with a small model for extraction today, that's the reason to try it. Also, on the Shopping task, all writers (including Opus 5) fail to reach 24 points, showing that some memory tasks can't be solved by swapping the writer.

Three Ways to Run Locally

llama.cpp / Ollama

Support for the hy_v3 architecture has entered mainline llama.cpp via PR #25395 and shipped with build b9993. Older versions cannot load it. The uploader's command uses the Q5_K_M quantized version, with key parameters including --jinja --chat-template-file and --chat-template-kwargs '{"reasoning_effort":"no_think"}'. With Ollama it's a one-liner: ollama run hf.co/rapatsky/Hy-Memory-30B-A3B-GGUF:Q4_K_M.

Two pitfalls to note: Tencent's config has moe_intermediate_size=1536 but the actual expert tensor is 768, which the uploader patched during conversion; community tests (34 English + 34 Russian dialogues) show all quantized versions can output valid JSON (68/68), but strict success is only 23/34, with failure modes being duplicated facts, over-storage, and occasional unfounded inferences—errors wrapped in a perfect JSON shell.

GGUF and K-quant quantization naming: GGUF is the single-file model container format used by llama.cpp, packing weights, tokenizer, and metadata together for easy layer-by-layer offloading in mixed CPU/GPU environments. The Q8_0, Q6_K, Q5_K_M, and Q4_K_M in filenames are llama.cpp quantization schemes: Q4_K_M means 4-bit, K-quant series, Medium mixed precision—most tensors at 4 bits, with some sensitive layers (such as attention output and embeddings) kept at higher bits to reduce precision loss. Smaller numbers mean smaller size and faster speed, but higher perplexity. The "Q4_K_M 18.2 GB" in the article is the result of this mixed quantization and the key to fitting on a 24 GB GPU. GGUF uploaders also often patch inconsistencies between the original config and actual tensor shapes, which is exactly the pitfall caused by the moe_intermediate_size discrepancy mentioned in the article.

vLLM (full BF16)

The official documentation path, tested on vLLM v0.31.0. vllm serve tencent/Hy-Memory-30B-A3B --served-model-name hy-memory starts it up, but this is the BF16 checkpoint with about 60 GB of weights, requiring one 80 GB card or two 48 GB cards with tensor parallelism.

Tensor Parallelism: vLLM's --tensor-parallel-size splits the weights of a single layer by dimension across multiple GPUs, with each card storing only a portion and aggregating results via All-Reduce during the forward pass. The article's "two 48 GB cards with tensor parallelism" means using TP=2 to split the roughly 60 GB BF16 weights across two cards, each holding about 30 GB of weights plus KV cache. Tensor parallelism requires high-bandwidth interconnect between cards (NVLink is best; PCIe works but throughput drops), and the number of cards usually needs to divide the number of attention heads evenly. For MoE models, vLLM also supports expert parallelism, placing different experts on different cards, but configuration is more complex. If you only have a single 80 GB card, loading BF16 directly with TP=1 is the simplest option.

MLX (Mac)

Upstream mlx-lm doesn't yet recognize hy_v3, but hermitdave's Mixed4-6 build includes hy_v3.py referenced by config, so it can be loaded directly with stock mlx-lm. The uploader reports about 65 tok/s decoding on an M3 Max 64 GB, perplexity within 0.94% of the 8-bit version, and 70/70 fact recall on extraction tests. Don't mistakenly download the -omlx variant—different expert keys will produce "Received 423 parameters not in model."

Connecting to Memory Frameworks

Tencent's Hy-Memory SDK (pip install hy-memory==2.0.3) accepts any OpenAI-compatible endpoint via MEMORY_LLM_BASE_URL and MEMORY_LLM_API_KEY; just point it at your llama-server or vLLM port. Be sure to use pro mode—lite mode doesn't call the LLM at all, so the writer never runs. You'll also need a separate embedding endpoint (MEMORY_EMBEDDER_MODEL, MEMORY_EMBEDDING_DIMS). Mem0 and MemOS likewise accept an OpenAI-compatible base URL.

The division of labor between OpenAI-compatible endpoints and embedding models: An OpenAI-compatible endpoint means the server implements interfaces like /v1/chat/completions, so clients only need to change base_url and api_key to use any local inference service (llama-server, vLLM, MLX's OpenAI-compatible layer) as if it were GPT. The Hy-Memory SDK's MEMORY_LLM_BASE_URL follows this path, so the writer model can be fully localized. But memory frameworks have a second type of call: converting text into vectors for similarity retrieval, i.e., embedding. MEMORY_EMBEDDER_MODEL and MEMORY_EMBEDDING_DIMS configure the embedding model and its dimensions; this is a separate endpoint from the writer—the writer decides "what to write," while the embedding model decides "how to retrieve." The article emphasizes that lite mode doesn't call the LLM because that mode skips writer inference and leaves only rules or embeddings.

Why Memory Belongs Outside the Model

Rather than baking memory into hidden states within the weights, I'd prefer a specialized model writing to a store I can inspect. When a user says "forget my address," I want to execute a DELETE and watch that row disappear. Hy-Memory-30B-A3B fits this approach: turning the expensive LLM call on every write into a cheap local operation, while memory always belongs to you.

What it doesn't solve is the ceiling on write quality. The 0.87-point lead over Opus 5 comes from vendor subsamples, and the community's strict pass rate of 23/34 says the same thing: before trusting it with user data, test it on your own conversations.

One-line Summary

Tencent open-sources a 30B MoE memory writer as a local replacement for GPT/Claude write calls, but its score surpassing Opus 5 is questionable.

Paragraph Summary

Tencent open-sourced Hy-Memory-30B-A3B, a mixture-of-experts model with 30B parameters and about 3B activated per token, positioned as a memory writer rather than a storage: it filters what information gets written to external memory and how old entries are updated, while the actual memory remains in vector databases in Mem0, MemOS, or Hy-Memory backends. The goal is to replace the expensive GPT/Claude call on every write with cheap local inference. The article covers the hardware threshold of the A3B architecture ("pay for memory at 30B, save on compute at 3B"; Q4_K_M 18.2 GB runs on a 24 GB GPU or 32 GB Mac), breaks down the noise problem behind the vendor benchmark's 0.87-point lead coming from a 20% subsample with per-item gaps of only 0.01–0.02 points, provides three deployment approaches (llama.cpp/Ollama, vLLM, MLX) and key configuration points for connecting to memory frameworks, and finally notes that the community's strict test pass rate is only 23/34—verify with your own conversation data before deploying.