HiCoMER: Hierarchical Memory for LLM Agents That Retrieves Only What's Still Valid

The Old Problem with Memory Retrieval: Semantic Relevance ≠ Current Validity

In multi-agent collaboration, memory is never static. Team memory carries collective decisions, collaboration protocols, and current consensus, while individual memory retains each member's observations, execution traces, and incremental progress. These two memory types differ in structure and evolve at different paces — yet most existing systems treat them exactly the same.

The dominant approach to memory-augmented systems pools all stored memories together and retrieves by semantic relevance, importance, or recency. This ignores the hierarchical structure of memory and its constantly shifting validity. The result: systems frequently surface memories that are semantically relevant but factually stale or contradictory — especially individual memories that no longer align with the team's current consensus.

This flaw is amplified when collaborative LLM agents need to answer user queries. Responses should be grounded in valid memory; retrieving outdated or conflicting content makes reliable output nearly impossible.

HiCoMER's Core Idea: Validate First, Then Retrieve

The HiCoMER framework, proposed in an arXiv paper, targets exactly this problem. Its key design insight is a process reordering: instead of retrieving directly from all stored memories, it first maintains the validity of team and individual memories, then retrieves only from those that remain valid.

This sequence shift is subtle but consequential. Traditional approaches treat all stored memories as candidates and rely on ranking to filter; HiCoMER performs a validity-based filtering pass before retrieval, removing expired or conflicting memories from the candidate pool entirely. Semantically relevant but outdated memories simply never get a chance to be surfaced.

Three Core Components

The framework consists of three modules:

• Hierarchical Memory Conflict Updater: Maintains the validity of both team and individual memories, and handles conflicts between individual memories and team consensus.
• Validity-Aware Memory Retriever: Retrieves exclusively from memories that remain valid, rather than the full memory store.
• Memory-Grounded Answer Generator: Ensures final responses are anchored to valid memories.

The three components form a complete pipeline — from conflict management through retrieval to generation.

The Hierarchical Memory Conflict Updater works by attaching a validity flag to each memory entry. When the team reaches a new consensus, the system scans related individual memories and marks any entries that contradict the new consensus as invalid — without deleting them. This soft-invalidation design preserves historical traceability while filtering invalid memories out of the retrieval candidate pool. The Validity-Aware Retriever then performs semantic matching on this filtered valid subset. Even if an individual memory is highly similar to a query, it will not appear in results if it has been flagged as conflicting with current team consensus. The full pipeline decouples "what memory still holds" from "which memory is most relevant" — solving each as a separate, independent step.

Hierarchical Modeling: Team Consensus Takes Priority Over Individual Traces

A key assumption in HiCoMER is that not all memories carry equal authority. Team memory represents the current collective consensus and holds higher epistemic weight; individual memory may contain intermediate states that haven't converged, or have since been overridden.

During collaborative evolution, an individual member's observations or execution outcomes may conflict with team decisions formed later. If the retrieval system has no awareness of this hierarchy, it will readily return stale individual memories as valid answers. The Hierarchical Conflict Updater's role is to dynamically adjust the validity status of individual memories whenever they conflict with team consensus — ensuring that current team decisions are never silently overwritten by outdated information.

This explicit modeling of structure and temporal evolution is what fundamentally differentiates HiCoMER from flat retrieval approaches.

Experimental Results: Fewer Stale Recalls, Higher QA Quality

To evaluate HiCoMER, the research team constructed two new datasets specifically designed for collaborative settings, centered on memory-grounded QA tasks. These datasets directly assess a system's ability to answer questions based on valid memories in a collaborative environment.

Across both datasets, HiCoMER consistently outperformed multiple strong baselines across three dimensions:

• Significantly reduced retrieval of outdated memories
• Better preservation of current team consensus
• Improved overall quality of downstream question answering

These results support the framework's core hypothesis: in settings where memory is heterogeneous and continuously evolving, validating first and retrieving second is more reliable than relying purely on semantic relevance ranking.

Memory-grounded QA is a task paradigm specifically designed to evaluate whether a system can correctly leverage stored memories when answering questions — distinct from open-domain QA, where answers must be traceable to specific entries in the memory store rather than relying on the model's pretrained parametric knowledge. In collaborative settings, the challenge intensifies because the memory store itself is dynamically evolving: the "correct answer" to the same question can change as team decisions are updated. The research team built new datasets rather than repurposing existing static QA benchmarks precisely to ensure the evaluation scenarios match real collaborative dynamics. The two datasets were designed to cover agent teams of varying scales and memory update frequencies, testing the framework's generalization across different collaboration rhythms.

Implications for Multi-Agent Systems

As LLM agents evolve from individual actors to collaborative teams, memory management is becoming an unavoidable challenge — both as an engineering problem and a research frontier. HiCoMER's value lies not only in its concrete technical solution, but in the perspective shift it proposes: the quality bottleneck in memory retrieval often isn't about finding relevant content — it's about whether the content found is still valid.

For developers building collaborative agent systems, this is a signal: when designing the memory layer, you need to introduce a validity dimension and hierarchical structure, rather than treating the memory store as a flat vector retrieval index. As team decisions continuously update and members independently advance their tasks, a memory mechanism capable of sensing conflicts and maintaining consensus may well become a critical factor in system reliability.