Keyframe Mnemonics: Solving the Memory Problem in Long-Horizon Behavior Cloning

Why Long-Horizon Behavior Cloning Is So Hard

Training policies in non-Markovian environments has long been one of robotics learning's toughest challenges. Non-Markovian means that the optimal action at any given moment depends not just on the current observation, but on contextual information accumulated over a long history. A robot completing a task may need to recall a critical cue seen much earlier and invoke it at exactly the right moment.

Existing approaches rely primarily on two architectural families to capture long-range dependencies — and both have significant shortcomings. Recurrent models are prone to hidden-state collapse and gradient instability, especially when trained via backpropagation through time (BPTT). Attention-based models, on the other hand, are fundamentally limited by context window length: once the window is exceeded, early information is simply forgotten.

In short, when task horizons stretch beyond what was seen during training, these models tend to fail quickly. This is the core problem this arXiv paper sets out to solve.

Behavior Cloning (BC) is the most fundamental paradigm in imitation learning: it directly learns a mapping from observations to actions from expert demonstration data, essentially framing the problem as supervised learning. Its strengths are simplicity and efficiency — no reward function design required. Its inherent weakness is compounding errors: once the policy drifts from the demonstration trajectory, it enters states never seen during training, and errors accumulate. As task horizons grow longer, this problem compounds with the long-range dependency challenge, making traditional BC methods nearly incapable of handling task sequences spanning minutes or more. Backpropagation Through Time (BPTT) is the standard algorithm for training recurrent neural networks. It requires unrolling the computational graph along the time axis to compute gradients — the longer the sequence, the higher the risk of vanishing or exploding gradients, which is the fundamental reason recurrent models perform poorly on ultra-long-horizon tasks.

The Core Idea Behind Keyframe Mnemonics

The research team proposes a self-supervised method called Keyframe Mnemonics. Its central insight is: rather than forcing a model to remember all historical observations, actively discover a small set of informationally critical observations — referred to in the paper as "mnemonics."

The approach works in two steps. First, a target function is learned by sampling from randomly selected historical observations and using it as a reward signal to filter which frames are worth remembering — this is the keyframe selection process. The entire procedure is self-supervised, requiring no manual annotation of which frames matter. Second, a behavior cloning (BC) policy is trained to model the action distribution conditioned on this discovered set of keyframes.

The elegance of this design lies in how it transforms "remember everything" into "remember the critical few." The policy's working memory maintains only a small set of decision-relevant keyframes at any given time — avoiding both the state collapse of recurrent models and the context-length bottleneck of attention-based models.

Theoretical Guarantees

The paper also provides theoretical backing: under certain task structure assumptions, the method offers context retention guarantees over unbounded time horizons. This means that regardless of how long a task runs, as long as the critical information is correctly captured as mnemonics, the policy can continuously access it without degradation over time. This is the origin of the "horizon-invariant" label in the paper's title.

The "horizon-invariant" property carries special value in robot learning. Most existing policies are trained on demonstrations of fixed length, and their generalization ability is typically tightly coupled to the training horizon — once the horizon exceeds the training distribution, performance degrades rapidly. This problem is particularly acute in sim-to-real transfer and long-horizon manipulation tasks: lab demonstrations tend to be short, edited sequences, while robots in real deployment face continuous, temporally open-ended interactions. Being able to prove that "no matter how long the horizon, critical information won't be lost due to temporal extension" means the horizon distribution shift between training and deployment is theoretically decoupled — a meaningful step toward improving real-world reliability.

Experimental Results: Generalization Is the Standout

The method was validated across two types of settings.

In synthetic memory task domains, the mnemonic-based BC policy achieved a 100% success rate and generalized to time horizons orders of magnitude longer than those seen during training with virtually no performance degradation. This is particularly significant — it directly demonstrates that the method's horizon-invariant property isn't just a theoretical result, but one that holds up in practice.

On more realistic memory-intensive robot manipulation benchmarks, the method achieved an average 13.9% absolute success rate improvement over the strongest baseline across 23 tasks. The most compelling result comes from real robot experiments: with time horizons extended to 20× the training length, the policy maintained a 80% success rate.

In contrast, traditional methods typically collapse when the time horizon is substantially extended. Keyframe Mnemonics maintains relatively stable performance — and this "train short, test long" generalization capability is enormously valuable in real deployment, where robots routinely face tasks much longer than any training demonstration.

Significance and Limitations

Keyframe Mnemonics offers a compelling new perspective: the long-horizon memory problem doesn't necessarily require larger context windows or more complex recurrent architectures. Actively selecting informationally critical keyframes may be the more elegant solution. It transforms "memory" from a passive architectural property into an explicit, learnable, and optimizable objective.

That said, its boundaries deserve honest appraisal. The paper's theoretical guarantees rest on "specific task structure assumptions," which means performance may be diminished in complex tasks where those assumptions don't hold. Additionally, the quality of mnemonic discovery depends heavily on the design of the self-supervised objective function — how robust this remains in more open-ended real-world environments still requires broader validation.

The team has made code and videos publicly available at keyframe-mnemonics.github.io. Researchers working on imitation learning, robot manipulation, and long-horizon decision-making will find this a worthwhile project to follow.