[KongchangAI]
· 2 min read· 1,157 words

HiVe-Mem: A Multi-Dimensional Memory Framework for More Efficient Multimodal Agents

HiVe-Mem: A Multi-Dimensional Memory Framework for More Efficient Multimodal Agents

HiVe-Mem boosts multimodal memory retrieval and cuts token usage by 90% via horizontal graph expansion and vertical evidence composition.

This article introduces HiVe-Mem, a multimodal memory framework addressing recall gaps and evidence rigidity in the "compress-retrieve" paradigm. It organizes memories into a relational graph with text and visual anchors for horizontal neighbor expansion, and lets a lightweight trainable selector compose query-adaptive evidence subsets from summaries, descriptions, visual grounding views, raw images, and interaction history. On three benchmarks, it consistently improves performance across base models while reducing token consumption by ~90% versus the most efficient baseline.

Multimodal agents accumulate a large amount of historical information during long-term interactions with users, and how to efficiently retrieve and utilize this information has become a key challenge. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which can lead to missed relevant memories and fail to adapt to the heterogeneous evidence needs of different queries. To address this problem, researchers propose HiVe-Mem—a multimodal memory framework that interleaves horizontal memory expansion with vertical evidence composition.

HiVe-Mem paper title screenshot

Limitations of Existing Memory Systems

Most current multimodal agent memory approaches follow a "compress-retrieve" paradigm: historical interactions are compressed into compact representations, and relevant segments are then retrieved via semantic similarity. The drawback of this approach is that the compression process loses detail, and semantic retrieval may also miss memories that are superficially dissimilar but actually complementary. In addition, different queries have very different needs for evidence forms—some questions require summary-level overviews, while others require fine-grained visual grounding or raw images. A single predefined representation struggles to satisfy this diversity.

HiVe-Mem's starting point is to simultaneously broaden the coverage of memory and enable each retrieved memory to flexibly contribute different forms of evidence according to query needs.

Semantic retrieval typically refers to encoding the query and memory segments into dense vectors separately, then finding the closest segments via measures such as cosine similarity. Its core assumption is that "semantically similar means relevant," but in long-term multimodal interactions, relevant memories may appear in different modalities or at different levels of abstraction. Proximity in vector space does not necessarily correspond to the contextual complementarity required by the task, so semantic retrieval alone is prone to recall blind spots.

Predefined representations mean that the system fixes the granularity and modality of memories at the storage stage (e.g., storing only summaries or only text descriptions), which cannot be changed at retrieval time. This rigid design creates a dilemma when query needs are diverse: representations that are too coarse lose detail, while representations that are too fine introduce redundancy and waste tokens.

Horizontal Memory Expansion: Recovering Complementary Information with a Relational Graph

HiVe-Mem organizes memory segments into a relational graph, where nodes are connected through text anchors and visual anchors. After semantic retrieval hits an initial seed memory, the system expands along the graph structure to its neighbor nodes, bringing memories that are semantically related but not directly retrieved into the candidate set. This process is called "horizontal expansion," and it compensates for the blind spots of pure semantic retrieval—memories that do not literally match the query but are contextually complementary can be recalled.

This graph structure design makes full use of multimodal information: text anchors capture conversation topics and entities, while visual anchors associate image content, and together they form the connective links between memories.

Text anchors and visual anchors can be understood as the "hooks" within memory segments used to establish associations. Text anchors are typically topic words, entity names, or event tags extracted from conversations; visual anchors come from objects, scenes, or regional features in images. When two memory segments share or are semantically associated through anchors, an edge is created between them in the relational graph. In this way, the graph structure encodes both language-level topical coherence and visual-level content relatedness, enabling neighbor expansion from seed nodes to recall complementary memories across modalities.

Vertical Evidence Composition: Tailoring an Evidence Subset for Each Memory

In the vertical dimension, HiVe-Mem allows each retrieved memory to retain multiple evidence forms, including compact summaries, descriptive text, visual grounding-based views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each memory, thereby adaptively leveraging heterogeneous multimodal information according to the specific query.

This means the same memory can present different "faces" when facing different questions: when a query requires macro-level understanding, the selector may favor summaries; when a query involves visual details, it may include raw images or grounding views. This query-adaptive evidence composition mechanism avoids the information loss or redundancy caused by uniform representations.

Visual grounding refers to mapping referring expressions in text descriptions to specific regions in an image—for example, locating "the red cup on the table" to a bounding box in the image. A visual grounding-based view uses this correspondence to crop or highlight the local image region relevant to the query, preserving key visual details while being more compact than the raw image. It sits between pure text descriptions and raw images, providing the selector with an evidence form that balances precision and efficiency.

Experimental Results: Performance Gains with Drastically Reduced Token Consumption

The researchers conducted experiments on three multimodal memory benchmarks. Results show that HiVe-Mem consistently improves task performance across different base models, while reducing token consumption by approximately 90% compared with the most efficient competitive multimodal baseline. This result indicates that through precise horizontal expansion and vertical evidence composition, the system not only sees more comprehensively but also uses less—significantly compressing inference costs while improving effectiveness.

The two dimensions of horizontal expansion and vertical composition work together: the former expands the recall coverage of memory, while the latter optimizes how each memory contributes evidence. Together, this two-dimensional design enables multimodal agents to both remember more and use memory more intelligently during long-term interactions.

Implications for Multimodal Agent Memory Design

HiVe-Mem's approach suggests that memory systems should not be merely passive storage and retrieval pipelines, but can actively construct associative structures between memories and endow memories with flexible expressive capabilities. Graph-based expansion alleviates the limitations of semantic retrieval, while multi-form evidence and a learnable selector respond to the heterogeneity of query needs. The drastic reduction in token consumption also shows that fine-grained evidence selection can replace coarse full-context injection, providing a viable path for cost control in long-term interaction scenarios.

For building multimodal applications that require long-term memory, this design—which balances breadth of coverage with evidence precision—is worth learning from.

One-line Summary

HiVe-Mem improves multimodal memory retrieval effectiveness and reduces token consumption by 90% through horizontal graph expansion and vertical evidence composition.

Paragraph Summary

This article introduces the multimodal memory framework HiVe-Mem. To address the recall omissions and evidence rigidity of the existing "compress-retrieve" paradigm caused by semantic retrieval blind spots and predefined representations, it proposes a two-dimensional design: in the horizontal dimension, memories are organized into a relational graph with text and visual anchors, and neighbor expansion from semantic retrieval seeds recalls complementary memories; in the vertical dimension, each memory retains multiple evidence forms such as summaries, descriptions, visual grounding views, raw images, and interaction history, with a lightweight trainable selector adaptively composing evidence subsets per query. On three multimodal memory benchmarks, the method consistently improves task performance across different base models and reduces token consumption by approximately 90% compared with the most efficient baseline, demonstrating that fine-grained evidence selection can replace full-context injection and providing an efficient memory solution that balances coverage and cost for long-term interaction scenarios.

#HiVe-Mem#multimodal agents#memory framework#relational graph#visual grounding#token efficiency
Share:

Related articles