METTA: Memory-Enhanced Meta-Reinforcement Learning for Test-Time Adaptation

Meta-RL aims to train agents that can quickly adapt to new tasks, but adaptation often requires extracting long-horizon task evidence from limited interactions. Classic methods such as recurrent neural networks (RNNs) compress history into a hidden state, while Transformers access past trajectories via attention mechanisms at the cost of quadratic memory and computation overhead. How to efficiently represent long interaction histories has become a key bottleneck for improving the scalability of meta-reinforcement learning.

A recent paper proposes a meta-reinforcement learning framework called METTA (Memory-Enhanced Test-Time Adaptation) that decomposes the timescales of fast adaptation to achieve effective long-term memory while maintaining linear complexity.

Core Idea: Dual-Timescale Memory Decomposition

METTA decomposes the fast adaptation phase into two timescales: a causal neural memory handles fine-grained within-episode adaptation and runs with linear sequence complexity, while a compact episode-level summary supports cross-episode long-horizon task inference without attending to the full interaction history. This design avoids the quadratic overhead of Transformers and compensates for the shortcomings of RNNs in long-range dependencies.

More critically, the introduction of neural memory gives the agent sufficient expressive capacity and test-time training capability. This means that during the meta-test phase, the agent can effectively adjust to unseen tasks or even task families, rather than merely memorizing patterns within the training distribution.

Causal neural memory refers to a class of external memory modules that are incrementally updated in a causal (autoregressive) manner, with typical implementations drawing on linear attention, state space models, or fast weights mechanisms. Unlike RNNs that compress all history into a single hidden state, neural memory maintains a set of readable and writable memory slots and performs write operations based on the current input at each time step, so that memory capacity grows with the number of parameters rather than sequence length. Since updates depend only on the current step and existing memory, computation and memory overhead grow linearly with sequence length; meanwhile, memory parameters can still be fine-tuned via gradient updates at test time, which is precisely why it possesses test-time training capability.

Test-time training refers to the model performing online updates on a subset of parameters during inference using unlabeled or self-supervised signals to adapt to the current input distribution. In meta-reinforcement learning, this means that when the agent faces a new task during the meta-test phase, it not only relies on forward inference but can also make gradient adjustments to the memory module based on a small number of interactions, thereby adapting to tasks outside the training distribution. Compared with traditional meta-learning that conditions solely through forward computation, test-time training provides stronger expressive capacity but also introduces trade-offs in computation overhead and stability.

Experimental Performance: Success Rate and Exploration Efficiency on Unseen Tasks

In unseen Meta-World ML10/45 test environments, METTA achieves a success rate exceeding 50% while maintaining a low training budget and short wall-clock runtime. This result is noteworthy because ML10/45 is the standard benchmark for evaluating generalization in meta-reinforcement learning; a success rate above 50% indicates that the method has practical adaptation capability on out-of-distribution tasks.

On the long-context sparse-reward PointMaze task, METTA exhibits more structured exploration behavior compared to Transformer-based baselines. Sparse-reward environments place extremely high demands on memory and exploration strategies, and METTA's ability to outperform Transformer baselines indicates that its memory mechanism maintains effective information organization over long sequences, rather than simply relying on the diffusion of attention weights.

Meta-World ML10/45 is a standard benchmark in the field of meta-reinforcement learning: ML10 contains 10 training tasks and 10 test tasks, while ML45 contains 45 training tasks and 5 test tasks. The test tasks are completely unseen during training and are used to evaluate the agent's generalization ability on out-of-distribution tasks. The tasks in this benchmark are all robotic manipulation skills (such as opening doors, pushing and pulling, pressing buttons, etc.), with unified state and action spaces for easy cross-comparison. PointMaze is a sparse-reward continuous control navigation task in which the agent must reach a goal point in a maze to receive a reward, and the observation does not directly indicate the goal direction, thus requiring the agent to possess long-horizon memory and structured exploration capabilities. It is often used to test the effectiveness of long-context memory mechanisms.

Technical Positioning and Implications

METTA's contribution lies in providing an efficient path: by combining linear-complexity neural memory with episode summaries, it simultaneously accommodates short-term adaptation and long-horizon inference. For scenarios such as robotic control and game AI that require rapid adaptation and involve high interaction costs, this design is expected to lower the training barrier and enhance deployment flexibility.

However, current experiments are concentrated on simulation benchmarks such as Meta-World and PointMaze. Whether the memory compression and summarization strategies remain effective in more complex, higher-dimensional real-world tasks still requires further validation. In addition, how the construction of episode-level summaries affects cross-task generalization is also worth exploring in future research.

Conclusion

METTA achieves efficient long-context adaptation in meta-reinforcement learning through dual-timescale memory decomposition. Its success rate on unseen tasks and structured exploration performance provide new ideas for scalable meta-reinforcement learning. With continued optimization of memory mechanisms, such methods are expected to play a role in a broader range of decision-making scenarios.

One-line Summary

METTA achieves efficient long-context adaptation in meta-reinforcement learning with linear complexity through dual-timescale memory decomposition.

Paragraph Summary

METTA is a meta-reinforcement learning framework designed to address the cost bottleneck of representing long interaction histories. It decomposes fast adaptation into two timescales: a causal neural memory handles fine-grained within-episode adaptation and runs with linear complexity, while a compact episode-level summary supports cross-episode long-horizon task inference, thereby avoiding the quadratic overhead of Transformers and compensating for the long-range dependency shortcomings of RNNs. Neural memory also endows the agent with test-time training capability, enabling adaptation to unseen tasks and even task families. It achieves over 50% success rate on Meta-World ML10/45 and exhibits more structured exploration behavior than Transformer baselines on the sparse-reward PointMaze task. This work provides an efficient path for scalable meta-reinforcement learning, but its effectiveness in real-world tasks remains to be validated.