REMORY
A paper published on arXiv proposing a method for compressing long conversational context using a minimal number of residual memory soft tokens. It uses sequences of 32 residual vectors to recover 8K-token conversation history at 95% fidelity, enabling context compression and memory augmentation for large language models.
Timeline (last 90 days)
REMORY (2026-10-08) proposes supplementing text-summary compression with a small set of learnable 'soft memory tokens,' achieving performance close to full context using only 5.2% of the input positions.
REMORY claims that a residual vector sequence of 32 dimensions can recover an 8K-token conversation with 95% fidelity.
REMORY's design completely decouples memory compression from LLM fine-tuning, without touching any weights of the LLM itself.
REMORY's encoder accepts raw history of up to 64K tokens and a human-readable summary of approximately 250 tokens, using a 12-layer Transformer with a 768-dimensional hidden state.
REMORY's residual decoder generates a bounded soft token sequence, each a 768-dimensional vector, stopping after N ≤ 64 tokens.
During training, REMORY uses KL divergence to measure the difference in output distributions between a frozen LLM seeing the full history and seeing the summary plus residual tokens.
REMORY's training data comes from 12TB of multi-turn conversations, with the summarization step performed by GPT-4-Turbo compressing each conversation to 250 tokens.
REMORY adopts a curriculum learning strategy, restricting N to 8 tokens in early epochs and gradually relaxing it to 64 in later epochs.
The REMORY paper proposes using a very small number of residual memory soft tokens to let a frozen LLM reconstruct long conversation history with nearly no loss.
On the 64K-token WebGPT-Long QA benchmark, REMORY raises BLEU-4 from 23.1 to 31.7, an increase of 37%.
7 more timeline events
All Facts (17)
REMORY (2026-10-08) proposes supplementing text-summary compression with a small set of learnable 'soft memory tokens,' achieving performance close to full context using only 5.2% of the input positions.
50%UnverifiedREMORY claims that a residual vector sequence of 32 dimensions can recover an 8K-token conversation with 95% fidelity.
50%UnverifiedREMORY's design completely decouples memory compression from LLM fine-tuning, without touching any weights of the LLM itself.
50%UnverifiedREMORY's encoder accepts raw history of up to 64K tokens and a human-readable summary of approximately 250 tokens, using a 12-layer Transformer with a 768-dimensional hidden state.
50%UnverifiedREMORY's residual decoder generates a bounded soft token sequence, each a 768-dimensional vector, stopping after N ≤ 64 tokens.
50%UnverifiedDuring training, REMORY uses KL divergence to measure the difference in output distributions between a frozen LLM seeing the full history and seeing the summary plus residual tokens.
50%UnverifiedREMORY's training data comes from 12TB of multi-turn conversations, with the summarization step performed by GPT-4-Turbo compressing each conversation to 250 tokens.
50%UnverifiedREMORY adopts a curriculum learning strategy, restricting N to 8 tokens in early epochs and gradually relaxing it to 64 in later epochs.
50%UnverifiedThe REMORY paper proposes using a very small number of residual memory soft tokens to let a frozen LLM reconstruct long conversation history with nearly no loss.
50%UnverifiedOn the 64K-token WebGPT-Long QA benchmark, REMORY raises BLEU-4 from 23.1 to 31.7, an increase of 37%.
50%UnverifiedOn the PersonaChat multi-turn dialogue benchmark, REMORY raises the exact match rate from 45.2% to 58.9% and reduces inference latency from 1.2 seconds per step to 0.68 seconds.
50%UnverifiedAll REMORY experiments use frozen Claude-3-Opus (175B) as the downstream model, and the MLPerf-LLM community has reproduced the results.
50%UnverifiedREMORY's cloud interface charges $0.001 per 1K residual tokens, delivering approximately 87% cost reduction compared to storing full history.
50%UnverifiedSoft tokens carry the risk of adversarial soft prompt attacks, where an attacker can craft vectors in the embedding space to manipulate the frozen LLM; norm clipping can serve as a mitigation measure but cannot fully eliminate the risk.
50%UnverifiedREMORY faces a token drift risk: once the data distribution shifts, performance degrades, requiring periodic fine-tuning of the encoder and monitoring of the exact match rate.
50%UnverifiedThe REMORY encoder adds approximately 120ms of latency on a single A100, which can be avoided by pre-computing residual tokens for predictable workflows.
50%UnverifiedREMORY converges after 250,000 training steps on 64 A100s; the final released checkpoint has approximately 350 million parameters and occupies 1.4GB of GPU memory.
50%