GMR
A training method for long-context recurrent memory models. It records the log-likelihood at the point when a fact is first introduced as an anchor point, and applies a hinge penalty to facts whose subsequent scores fall beyond a tolerance boundary, in order to preserve the ability to retrieve memorized facts.
Timeline (last 90 days)
An engineering student recently posted a research proposal on Reddit aimed at addressing the "state-induced forgetting" problem of recurrent compressed memory in long-context LLMs, including a training method GMR and an evaluation protocol STRATA.
The GMR training objective records the model's length-normalized log-likelihood of the correct answer when a fact is first introduced as an anchor, and after subsequent memory updates applies a penalty if the score falls below the anchor by more than a tolerance margin, with the retention loss formula being L_ret = (1/B) * sum_f max(0, anchor_f - margin - current_score_f).
For sampled facts whose scores fall below the anchor by more than the margin, the gradient of GMR's hinge loss is the same as the gradient of replaying the correct answers to those facts; facts that have not degraded produce no retention gradient.
The proposal plans to compare five configurations: a baseline with no explicit retention objective, uniform replay, loss-prioritized replay, output distillation from a full-context reference, and the GMR objective, compared under matched memory size, training stream, and replay budget.
The author publicly poses five questions on which feedback is sought, including whether the problem formulation is useful, whether distinguishing GMR from ordinary replay is scientifically meaningful, whether STRATA reveals failure modes better than existing evaluations, whether the experimental comparison is fair, and which relevant literature has been missed.
All Facts (5)
The GMR training objective records the model's length-normalized log-likelihood of the correct answer when a fact is first introduced as an anchor, and after subsequent memory updates applies a penalty if the score falls below the anchor by more than a tolerance margin, with the retention loss formula being L_ret = (1/B) * sum_f max(0, anchor_f - margin - current_score_f).
50%UnverifiedFor sampled facts whose scores fall below the anchor by more than the margin, the gradient of GMR's hinge loss is the same as the gradient of replaying the correct answers to those facts; facts that have not degraded produce no retention gradient.
50%UnverifiedThe proposal plans to compare five configurations: a baseline with no explicit retention objective, uniform replay, loss-prioritized replay, output distillation from a full-context reference, and the GMR objective, compared under matched memory size, training stream, and replay budget.
50%UnverifiedAn engineering student recently posted a research proposal on Reddit aimed at addressing the "state-induced forgetting" problem of recurrent compressed memory in long-context LLMs, including a training method GMR and an evaluation protocol STRATA.
50%UnverifiedThe author publicly poses five questions on which feedback is sought, including whether the problem formulation is useful, whether distinguishing GMR from ordinary replay is scientifically meaningful, whether STRATA reveals failure modes better than existing evaluations, whether the experimental comparison is fair, and which relevant literature has been missed.
50%