Personalized Memory Gating under Delayed Feedback: A Warm-Start UCB Approach

For language model agents to truly "remember" users, long-term memory is indispensable. But memory storage is limited, so it is impossible to save every extracted fact. Existing memory gating strategies typically treat all users the same, ignoring differences in how different users reuse information in the future. A new paper refines memory gating to the fact level and proposes an online learning framework based on delayed one-sided feedback, enabling agents to learn to decide in a "personalized" way for each user which memories are worth keeping.

Background: Why Memory Gating Needs Personalization

Long-term memory allows language model agents to maintain coherence and personalization across subsequent conversations, but memory storage is limited. When an agent extracts a large number of facts from interactions, it must decide which to write to long-term memory and which to discard—this is memory gating.

Existing approaches often adopt a single globally shared gating policy, assuming all users have the same information reuse patterns. In reality, however, different users care about vastly different things: some want the agent to remember project details, while others care more about personal preferences. A shared policy cannot capture this user specificity, leading to wasted memory resources or loss of key information. The paper therefore moves the problem down to the fact level and learns a separate gating policy for each user.

Core Method: Learning from Delayed One-Sided Feedback

Automatic Labeling Based on Retrieval

To train a gating policy, we first need to know whether each fact is "useful." The paper proposes a retrieval-based labeling strategy: for each extracted fact, observe whether it is retrieved and incorporated into a response in subsequent interactions. If so, label it positive; otherwise, label it negative. This labeling method requires no human intervention and directly uses the agent's own retrieval behavior as the supervision signal.

Modeling Gating as a Contextual Apple Tasting Problem

Online memory gating faces two tricky characteristics: feedback is delayed (fact reuse may occur many turns later), and only facts that are kept can be observed as reused or not (one-sided feedback). The paper formalizes this as a contextual apple tasting problem with delayed one-sided feedback—the agent chooses each turn whether to keep a fact, but only when it chooses "keep" can it obtain feedback on whether that fact is reused.

Contextual apple tasting originates from the "apple tasting" framework in online learning: each round, the decision-maker chooses between "accept" or "reject" an item, but only when choosing "accept" can the true label (good or bad) of the item be observed; when rejecting, no feedback is available, so feedback is one-sided. In the memory gating setting, "accept" corresponds to "keep the fact" and "reject" corresponds to "discard the fact"; only kept facts may be retrieved later, thereby revealing whether they are reused. Adding "context" means each round's decision is also accompanied by a feature vector (such as user profile, fact type, conversation history, etc.), and the algorithm needs to choose actions based on context, which extends the problem from a simple multi-armed bandit to a contextual bandit, significantly increasing learning difficulty and sample complexity.

Warm-Start UCB Algorithm

To address the above problem, the authors propose Warm-Start UCB: first use existing offline data to learn an initial policy, then continuously adjust it during online interaction with the current user. This "warm start" combines the sample efficiency of offline learning with the personalization capability of online adaptation. Theoretical analysis shows that when the delay window does not exceed O(√T), the algorithm's high-probability regret upper bound is Õ(Tε ∧ d√T), where T is the total number of rounds, ε is the initialization error, and d is the feature dimension.

UCB (Upper Confidence Bound) is a classic multi-armed bandit algorithm whose core idea is to "face uncertainty optimistically": for each action's estimated expected reward, add a confidence upper bound term and prioritize the action with the highest upper bound, thereby balancing exploration and exploitation. In contextual bandits, UCB is usually combined with a linear model (LinUCB), using feature vectors to estimate rewards and compute confidence intervals. Warm-Start refers to using offline data to pretrain an initial policy or initialize model parameters in advance, rather than learning online from scratch. This can greatly reduce the exploration cost of the online phase, especially in difficult settings with delayed and one-sided feedback, where the prior knowledge provided by warm start can accelerate convergence and ensure that the initialization error term ε in the regret upper bound is controllable.

Experimental Results and Implications

Experimental results show that under different memory budgets and history lengths, Warm-Start UCB outperforms all baseline methods. This indicates that personalized gating can indeed improve memory utilization efficiency, and that the combination of offline warm start and online adaptation is effective.

For agent developers, the value of this work lies in providing an actionable framework: automatically label using retrieval behavior, and make decisions using online learning with delayed feedback. It also reminds us that the design of memory systems should not stop at "what to store," but should pay more attention to "for whom to store it"—personalized gating may be a key step for long-term memory agents to become practical.

One-line Summary

The paper proposes fact-level personalized memory gating and a Warm-Start UCB online learning framework to improve the long-term memory efficiency of agents.

Paragraph Summary

This article addresses the gating problem in long-term memory for language model agents. It points out that existing globally shared policies cannot adapt to differences in how different users reuse information, so it refines gating to the fact level and learns a retention policy separately for each user. The authors use whether a fact is retrieved in subsequent interactions as an automatic supervision signal and formalize the online decision as a contextual apple tasting problem with delayed one-sided feedback, proposing the Warm-Start UCB algorithm: first warm-start with offline data, then adapt online. Theoretical analysis gives a regret upper bound of Õ(Tε ∧ d√T), and experiments show that the method outperforms baselines under different memory budgets and history lengths, providing an actionable framework for personalized long-term memory systems.