Multi-User Multi-Agent Memory Extraction: How MixUA Solves the Sharing and Privacy Dilemma

Multi-agent systems are moving from solo work to group collaboration, but a key contradiction is often overlooked: when agents learn from each other's experiences, how do you ensure each piece of experience is only seen by those who should see it? The MixUA system from academia attempts to answer this question.

The Memory Dilemma of Self-Evolving Agents

Self-evolving agents can improve skills, memory, and tool code from experience without retraining the base large model. Group evolution goes a step further, letting agents share lessons learned with one another. However, most existing systems assume "sharing is beneficial" by default, ignoring two fundamental questions in multi-user, multi-agent scenarios: Who is this experience useful for? What information must remain private?

For example, User A teaches an agent a code debugging preference. This preference may also help User B, but User A's account information must never leak because of it. Similarly, tricks that Agent X accumulates during tasks may apply to Agent Y, but indiscriminate global sharing could bring a specific user's private facts into public memory.

Self-evolving agents are a class of agents that continuously improve their capabilities by accumulating and reusing experience without retraining the base large model. Their core mechanisms typically include: distilling successful strategies, failure lessons, user preferences, or tool-calling code from past tasks into retrievable memory entries, then recalling, combining, and reusing them on demand in subsequent tasks. This "experience memory" can be divided into skill memory (how to perform a certain type of operation), fact memory (knowledge about users or the environment), and tool memory (reusable code snippets). Group evolution builds on this by letting multiple agents share these memory entries with one another, thereby accelerating individual learning. However, if the sharing granularity of memory is uncontrolled, skill benefits and privacy risks spread together—this is precisely the key contradiction that needs governance in multi-user multi-agent scenarios.

MixUA: Allocating Memory Granularity by Benefit Scope

MixUA's core idea is to distill independent user-agent interaction trajectories into "lessons learned," then assign each lesson to one of four scopes based on who benefits and disclosure restrictions: user-agent pair, single user, single agent, or global.

The key to this multi-granularity design is that the system does not simply stuff all memories into one global pool. Instead, it first determines who a piece of experience is valuable to and whether it contains sensitive information, then decides its visibility scope. This preserves the benefits of cross-user and cross-agent sharing while drawing boundaries for privacy.

Scope here refers to the visibility and recall range of a memory entry—the concrete implementation of access control policy in a memory system. The four scopes MixUA defines form a hierarchy from narrow to wide: the user-agent pair scope is visible only to a specific user-agent combination and is the most private; the user scope is visible to all agents that a given user interacts with; the agent scope is visible to all users served by that agent; and the global scope is open to all users and agents. Which scope a lesson is assigned to depends on two judgments: who it benefits (who has reuse value for it) and what its disclosure restrictions are (whether it contains sensitive facts that identify a person or account). This design—categorizing by benefit scope rather than simply by source—enables the system to balance shared benefits and privacy boundaries.

CrossLearnBench: A Hybrid Benchmark with Privacy Probes

To evaluate such systems, the research team proposed CrossLearnBench, a hybrid user-agent benchmark. It pairs existing agent tasks with user preferences and privacy probes, specifically testing whether private facts leak into memory readable by other users.

The experimental setup includes five users and seven agents. Results show that compared with a no-memory baseline, MixUA improves task pass rate by up to 16 percentage points; compared with a single global memory, it improves by 5 percentage points. Interestingly, MixUA still delivers gains even when a task has previously been attempted only by other users on other agents. In terms of preference retention, it preserves up to 90% of previously stated preferences; on privacy, at most only 0.02% of privacy probes find implanted private facts in forbidden scopes.

Privacy probes are controlled test signals used in the benchmark to detect memory leakage. Specifically, a uniquely identifiable private fact (such as specific account information or a fictional personal detail) is implanted in a user-agent interaction, and then memory is retrieved in scopes where that fact is forbidden to appear, checking whether other users or agents can read it. If the probe is recalled in a range where it should not appear, it is judged as a leak. Because probe facts are artificially constructed and precisely trackable, this method turns "whether privacy leaks" into a quantifiable metric—the statement in the text that "at most 0.02% of privacy probes find implanted private facts in forbidden scopes" refers to this proportion. Pairing privacy probes with user preferences also allows simultaneous evaluation of the system's performance in sharing useful experience and maintaining private boundaries.

Implications for Multi-Agent Collaboration

MixUA's significance goes beyond being just a memory system. It points to a direction: scaling multi-user, multi-agent systems requires fine-grained memory governance rather than crude global sharing. By binding lessons learned to benefit scopes and disclosure permissions, the system can achieve cross-user and cross-agent collaborative evolution without sacrificing privacy.

Of course, the current evaluation is still limited to five users and seven agents; the number of users, heterogeneity of agents, and complexity of privacy regulations in real scenarios will bring greater challenges. But MixUA provides a scalable path that lets users and agents share experience while each retaining their boundaries.

One-line Summary

MixUA allocates agent memory scopes by benefit range and disclosure restrictions, balancing shared benefits and privacy.

Paragraph Summary

The article introduces a multi-user multi-agent memory system called MixUA, aimed at resolving the privacy-versus-sharing contradiction of "who should see this experience" when agents learn from one another. Existing systems mostly assume sharing is beneficial, ignoring the two questions of who experience is useful for and what information must remain private. MixUA distills independent user-agent interaction trajectories into lessons learned, assigning them to one of four scopes—user-agent pair, single user, single agent, or global—based on benefit targets and disclosure restrictions. The research team also proposed the CrossLearnBench benchmark with privacy probes, tested under a setup of five users and seven agents. Results show that compared with a no-memory baseline, MixUA improves task pass rate by up to 16 percentage points, and compared with a single global memory, by 5 percentage points; it preserves up to 90% of previous preferences, and at most 0.02% of privacy probes find private facts in forbidden scopes. The article argues this points to the direction that multi-agent systems need fine-grained memory governance.