MemT-1.0: Memory-Native Evolution That Gives Models Built-In Memory

Memory is shifting from an external add-on to an intrinsic capability of the model. MemT-1.0 uses architectural changes to let the base model itself accumulate and retrieve information, rather than relying on retrieval augmentation or ultra-long context.

Why memory needs to be "native"

There are two mainstream approaches to getting models to remember information today: one is expanding the context window and stuffing all history into the prompt; the other is retrieval-augmented generation (RAG), which looks up relevant snippets from an external store. Both treat memory as something appended to the model, which brings problems such as high inference cost, bloated persistent storage, and low information utilization.

MemT-1.0 starts from a different premise: it treats memory as part of the base model's computation. In the authors' words, memory-native evolution requires "making memory an intrinsic capability of the base model," enabling it to continuously accumulate and leverage information. The key to this idea is that the model no longer merely processes input passively, but actively updates its internal memory state at every turn of interaction.

Retrieval-augmented generation (RAG) is a paradigm that decouples the model from an external knowledge base: the system first retrieves text snippets relevant to the current input from external storage, then stitches these snippets into the prompt for the model to generate. RAG's advantage is that knowledge can be updated at any time without retraining the model, but each inference incurs extra retrieval and stitching, with issues such as retrieval latency, snippet-selection errors, and context occupation, and the model itself does not truly "remember" this information.

KV cache refers to the key-value pairs cached by an autoregressive Transformer during generation, used to avoid recomputing attention over historical tokens. The longer the context, the more VRAM the KV cache occupies, and it grows linearly with the number of conversation turns—this is precisely the main source of VRAM usage and latency in long-context deployment. The "66x storage reduction" claimed by MemT-1.0 is measured against the scheme that retains the full historical KV cache.

Architectural core: Gated DeltaNet-2 and Mixture-of-Storage

MemT-1.0 builds on Metis, adopts Gated DeltaNet-2, and introduces a stable token-level memory update mechanism within each turn. This means memory writes happen at token granularity and the update process is stable—avoiding memory state drift or forgetting over long sequences.

To expand native memory capacity, the team further designed Mixture-of-Storage. You can think of it as partitioning different levels of storage units inside the model, allocating and retrieving them on demand, thereby adding usable memory space beyond the parameter count. This design directly addresses the bottleneck of "limited native memory capacity."

Two-stage training pipeline

After careful backbone preparation, MemT-1.0 adopts two-stage training: continued pre-training plus post-training. The entire pipeline is supported by 34B tokens of training data, along with efficient training and inference infrastructure. The combination of training data scale and infrastructure is the prerequisite for the memory mechanism to converge stably.

DeltaNet belongs to a class of linear attention/state space models, whose core is to replace the full-history KV storage in standard attention with an updatable recurrent state matrix, writing new information incrementally into the state in a "delta" manner each time. Compared with standard attention, its computation and storage grow linearly rather than quadratically with sequence length. Gated refers to introducing a learned gating mechanism that controls the proportion of new versus old information written and retained, thereby mitigating state drift and forgetting over long sequences—this is also the technical source of the "stable token-level memory update" emphasized in the text.

Mixture-of-Storage borrows the sparse routing idea of Mixture-of-Experts (MoE): the model maintains multiple groups of storage units internally, and a routing mechanism dynamically selects which part to activate based on the input, thereby expanding usable memory capacity without proportionally increasing per-token computation. Its difference from MoE is that what is routed is memory/storage slots rather than feed-forward network experts.

Continued pre-training refers to continuing next-token prediction training on a base model that has completed general pre-training, using domain- or objective-specific data, to inject new capabilities without training from scratch. In MemT-1.0, the role of this stage is to teach the model to write and update internal memory states at token granularity. Post-training typically includes steps such as supervised fine-tuning and preference alignment, used to equip the model with instruction following and specific task behaviors. The two stages combine to first shape the memory mechanism and then align how it is used, while the 34B-token scale is the data foundation ensuring convergence of this mechanism.

Model family and measured performance

MemT-1.0 released two versions: MemT-1.0-Nano and MemT-1.0-Flash. The team conducted a comprehensive evaluation of both memory capability and general capability.

Results show that both models on average outperform parametric memory baselines on memory-related tasks, remain competitive against retrieval-augmented methods, and largely preserve general capability. This is worth noting: many memory-enhanced schemes sacrifice general capability, whereas MemT-1.0 strikes a balance between the two.

Efficiency data: 1.86x speedup and 66x storage reduction

At a history length of 128K tokens, MemT-1.0-Nano achieves about 1.86x end-to-end inference speedup, with persistent storage reduced 66x compared with a full-context KV cache. These two numbers map directly to pain points in real deployment—VRAM usage and latency brought by long context. Compressing history into native memory states rather than retaining all KVs is the source of the efficiency gain.

Agentic tasks: internalizing procedural experience

MemT-1.0 also improves on agentic tasks. By internalizing retrieved procedural experience, the model's success rate on agentic tasks improves. This means memory is not just about remembering facts, but also about remembering "how to do" experience and reusing it in subsequent tasks.

What the memory-native route means

The value of MemT-1.0 lies not in刷新ing any single metric, but in validating a path: memory can be trained as part of the model architecture rather than stitched on afterward. For developers, this means that when deploying long-horizon dialogue and continuously learning agents in the future, it may no longer be necessary to maintain massive external memory stores and complex retrieval pipelines.

Of course, native memory also brings new questions: how memory is updated, how it is forgotten, and how to ensure no erroneous information is introduced—these are not yet fully addressed in the paper. But as "a step toward memory-native evolution," MemT-1.0 offers a reproducible engineering solution and comparable quantitative results, which is itself worthy of attention.

One-line Summary

MemT-1.0 internalizes memory as an architectural capability of the model, achieving speedup and storage reduction under long context.

Paragraph Summary

MemT-1.0 proposes a "memory-native" route, arguing that memory should be part of the base model's computation rather than relying on external add-ons such as ultra-long context or retrieval augmentation. Its architecture builds on Metis, uses Gated DeltaNet-2 to achieve stable token-level memory updates, and uses Mixture-of-Storage to expand native memory capacity. Training follows a two-stage pipeline of continued pre-training plus post-training, supported by 34B tokens of data. The team released Nano and Flash versions, which measured better than parametric memory baselines on memory tasks, remain competitive with retrieval-augmented methods, and largely preserve general capability. At 128K history length, Nano achieves about 1.86x end-to-end inference speedup and 66x persistent storage reduction, and improves agentic task success rates by internalizing procedural experience.