AdaMem: A Soft Compression Method That Intelligently Allocates Memory Tokens by Relevance in RAG

The Efficiency Dilemma in RAG: The Cost of Long-Context Compression

Retrieval-Augmented Generation (RAG) enhances language model performance by incorporating externally retrieved evidence, and has become the dominant paradigm for building knowledge-intensive applications. But this mechanism comes with an unavoidable challenge: when a model needs to process large volumes of lengthy passages, computational costs escalate sharply, and redundant or irrelevant information can interfere with generation quality, causing answers to drift off-topic.

To address this pressure, researchers have proposed "soft compression" — encoding retrieved passages into a compact set of continuous memory embeddings before generation, replacing verbose text with a small number of vectors. This approach significantly reduces inference overhead, but existing methods share a glaring design flaw.

"Soft compression" and "hard compression" represent two fundamentally different paths for context compression. Hard compression operates in token space, shortening text through extraction, pruning, or summarization — the result remains readable natural language. Soft compression operates in vector space, encoding entire passages into a handful of dense continuous embedding vectors. These vectors cannot be read directly by humans, but can be passed into a language model's attention layers as "soft prompts," allowing the model to reason directly over vector representations. The advantage of soft compression lies in its extremely high information density — in theory, a few vectors can capture the semantics of hundreds of tokens — but the trade-off is that it demands tighter alignment training between the compressor and generator, and the compressed representations are not portable across models. OSCAR is a pioneering representative of this field, validating the feasibility of replacing retrieved passages with a fixed number of memory embeddings. Its default "uniform allocation" design also makes it the most direct baseline that AdaMem is measured against.

The Waste of Uniform Allocation: The Shortcomings of Existing Soft Compression

The core issue lies in the allocation strategy. Existing soft compression methods (such as OSCAR) typically assign the same number of memory embeddings to each retained passage, with no consideration of how relevant that passage is to the current query.

This is clearly wasteful. In real-world retrieval scenarios, passage quality in the retrieval pool varies widely — some passages closely match the question and contain key answers, while others are only tangentially related or completely irrelevant. Allocating the same "memory budget" to both types of passages both prevents important information from being fully represented and wastes precious token quota on irrelevant content. This inefficiency of uniform allocation is especially pronounced when the retrieval pool is large and the available memory budget is tight.

AdaMem's Core Idea: Relevance-Guided Dynamic Allocation

To address this pain point, researchers propose AdaMem — a relevance-guided soft compression framework. Its core innovation is mapping learned passage relevance estimates to query-aware dynamic allocation of a fixed memory token budget.

In terms of concrete mechanisms, AdaMem has several noteworthy design choices:

Compression and Scoring in a Single Forward Pass

AdaMem uses a shared, query-conditioned compressor that produces two outputs in a single forward pass: continuous passage memory representations, and a relevance score for each passage. This means relevance estimation adds almost no extra overhead, avoiding the two-stage cost of "score first, then compress."

Deterministic Budget Allocation Rule

Once relevance scores are obtained, AdaMem uses a deterministic allocation rule to distribute memory tokens: passages with higher scores receive more memory tokens, while passages with scores below a threshold can be completely omitted. This focuses the limited memory budget on truly valuable evidence — putting resources exactly where they matter most.

The design motivation behind the "query-conditioned compressor" is significant: traditional compressors typically take only the passage itself as input, generating query-agnostic general summary representations. But in RAG scenarios, the informational value of the same passage can vary dramatically across different questions — a passage about a city's climate is extremely important when answering "how much snow does this city get in winter," but nearly useless when answering "what is this city's GDP." Injecting the query signal into the compression process allows the compressor to "know who it's serving," enabling it to prioritize semantic details most relevant to the current question within the limited memory vectors. This idea aligns with "query-aware summarization" in information retrieval — the core principle being that compression should not pursue general fidelity, but rather maximize information utility for a specific query.

Experimental Results: The More Aggressive the Compression, the Greater the Advantage

The research team evaluated AdaMem on six open-domain question answering benchmarks. AdaMem consistently outperforms OSCAR (the tight comparison baseline using uniform allocation) and other soft compression methods under the same memory budget.

A few key figures are particularly telling:

• At the standard 16× compression ratio, AdaMem's sub-string match improves by up to 3.2 percentage points (5.5%) over the uniform allocation baseline, with an average relative gain of 3.4%.
• At the more aggressive 64× compression ratio, the average relative gain expands to 14.6%, with the highest improvement reaching 9.8 percentage points (19.7%) on the PopQA dataset.

This trend is revealing: the higher the compression ratio and the tighter the budget, the greater the advantage of intelligent allocation over uniform allocation. The reason is straightforward — when every token becomes extremely scarce, the decision of "who gets the resources" is multiplied in value.

Sub-string match is a widely adopted automated metric in open-domain QA evaluation: a model's generated answer is considered correct if it contains the gold answer string as a substring. Compared to exact match, this metric is more lenient toward different phrasings and can accommodate minor wording variations; compared to semantic similarity-based evaluation, it is simpler and more reproducible. PopQA is a QA dataset built around long-tail Wikipedia entities, characterized by questions that involve obscure knowledge where the probability of a language model answering correctly from parametric memory alone is low — making it more dependent on retrieval quality. This explains why AdaMem's improvement on PopQA (19.7%) far exceeds the average: when the model itself "doesn't know the answer," whether retrieval information is sufficiently preserved has a multiplied impact on the final result.

Efficiency: Quality Improvements Without Sacrificing Speed

Performance gains lose practical value if they come at the cost of efficiency. AdaMem delivers a solid answer here: it maintains an efficiency profile comparable to the uniform compression baseline, while reducing inference latency by up to 4× compared to full-context inference.

More critically, AdaMem can match the answer quality of uncompressed approaches while operating at a latency as low as 1/4 of the full-context baseline. In other words, it speeds up inference without meaningfully degrading answer accuracy — precisely the ideal outcome that soft compression technology strives for.

Conclusion: Making Smart Trade-offs Under a Scarce Budget

AdaMem's value proposition can be summarized in one sentence: relevance-guided memory allocation is especially effective when the retrieval pool is large and the memory budget is tight. Rather than introducing complex additional modules, it achieves smarter resource allocation through a clean design — a shared compressor that produces results in one pass, paired with a deterministic allocation rule.

For teams building large-scale RAG systems who are sensitive to inference cost and latency, the "allocate memory on demand" approach that AdaMem provides is worth paying attention to and incorporating into real pipelines. As RAG applications continue to scale, maximizing the information density of retrieved evidence under a limited budget will become an increasingly central engineering and research challenge.