Keyframe Mnemonics
A self-supervised method for long-horizon behavior cloning that automatically discovers and memorizes keyframes (mnemonics) to address long-range dependency problems. It enables horizon-invariant policy learning without requiring manual annotation.
Timeline (last 90 days)
Keyframe Mnemonics is a self-supervised method whose core is to actively discover a small set of informationally critical observations as "mnemonics," rather than memorizing all historical observations.
Keyframe Mnemonics consists of two steps: the first step selects keyframes by learning an objective function from randomly sampled historical observations and using it as a reward signal; the second step trains a BC policy to model the action distribution based on the keyframes.
The paper provides a theoretical guarantee that, given certain task structure assumptions, the method can provide context-preservation guarantees over unlimited time horizons.
In a synthetic memory-task domain, the mnemonic-based BC policy achieved a 100% success rate and generalized to time horizons orders of magnitude longer than those seen during training.
On memory-intensive robotic manipulation benchmarks, the method achieved a 13.9% average absolute improvement in success rate over the strongest baseline across 23 tasks.
In real-robot experiments, the policy maintained an 80% success rate under conditions where the time horizon was extended to 20 times that of training.
The theoretical guarantees of this method rest on specific task structure assumptions, and its effectiveness may be diminished in complex tasks where these assumptions are not satisfied.
The quality of mnemonic discovery is highly dependent on the design of the self-supervised objective function.
The research team has publicly released code and videos on the project page keyframe-mnemonics.github.io.
All Facts (9)
Keyframe Mnemonics is a self-supervised method whose core is to actively discover a small set of informationally critical observations as "mnemonics," rather than memorizing all historical observations.
50%UnverifiedKeyframe Mnemonics consists of two steps: the first step selects keyframes by learning an objective function from randomly sampled historical observations and using it as a reward signal; the second step trains a BC policy to model the action distribution based on the keyframes.
50%UnverifiedThe paper provides a theoretical guarantee that, given certain task structure assumptions, the method can provide context-preservation guarantees over unlimited time horizons.
50%UnverifiedIn a synthetic memory-task domain, the mnemonic-based BC policy achieved a 100% success rate and generalized to time horizons orders of magnitude longer than those seen during training.
50%UnverifiedOn memory-intensive robotic manipulation benchmarks, the method achieved a 13.9% average absolute improvement in success rate over the strongest baseline across 23 tasks.
50%UnverifiedIn real-robot experiments, the policy maintained an 80% success rate under conditions where the time horizon was extended to 20 times that of training.
50%UnverifiedThe theoretical guarantees of this method rest on specific task structure assumptions, and its effectiveness may be diminished in complex tasks where these assumptions are not satisfied.
50%UnverifiedThe quality of mnemonic discovery is highly dependent on the design of the self-supervised objective function.
50%UnverifiedThe research team has publicly released code and videos on the project page keyframe-mnemonics.github.io.
50%