LUMIR: Decoupled Multimodal Representation Learning for Improved POI Recommendation

LUMIR uses LLMs to unify modality semantics and contrastive learning to decouple representations from downstream models, improving recommendation representation reusability.
This article introduces the multimodal recommendation framework LUMIR, which addresses the problem that existing methods tightly couple multimodal fusion with downstream models, resulting in representations that lack generality. LUMIR works in three steps: first, it uses large language models to map heterogeneous information such as images and text into a shared text space to eliminate semantic gaps; then it models cross-modal interactions through an interaction-aware fusion mechanism and introduces sequential signals as structural constraints into representation learning; finally, it separates representation learning from downstream recommendation optimization so that representations can be trained and reused independently. On a multimodal POI dataset under the next-POI setting, LUMIR's learned representations consistently outperform existing multimodal fusion baselines on multiple sequential recommendation backbones across both accuracy and beyond-accuracy metrics. The code is open-sourced, and the article argues this reflects a trend from end-to-end tight coupling toward modular, reusable representation learning.
A New Approach to Multimodal Recommendation: Decoupled Representation Learning and Downstream Models
Multimodal tourism information is becoming increasingly rich, and how to effectively leverage this heterogeneous data to improve recommendation performance has become a research hotspot. However, existing methods typically tightly couple multimodal fusion with the architecture and optimization objectives of specific downstream models, resulting in learned representations that lack generality and reusability. To address this issue, researchers propose the LUMIR framework, which decouples multimodal representation learning from downstream optimization, providing a new paradigm for learning reusable multimodal representations.

Core Design of the LUMIR Framework
LUMIR is a decoupled multimodal representation learning framework based on contrastive learning. Its core approach consists of three steps:
Semantic Unification: LLM-Driven Mapping to Text Space
First, multimodal information collected from heterogeneous data sources (such as images and text) is semantically unified through large language models (LLMs) and mapped into a shared text representation space. This step eliminates the semantic gap between different modalities and lays the foundation for subsequent fusion.
Why use text space as the shared representation space? Semantic alignment across modalities such as vision and language typically relies on large-scale paired data or dedicated cross-modal encoders, which are costly to train and difficult to generalize to new domains. Large language models have already absorbed massive textual knowledge during pretraining and possess a fairly thorough semantic understanding of heterogeneous information such as image descriptions and item attributes. Therefore, by using LLMs to transcribe content from each modality into natural language descriptions and then uniformly embedding them with a text encoder, the adaptation challenges of modality-specific encoders can be bypassed. The cost of this approach is that information may be compressed or lost during the "transcription" process (e.g., fine-grained visual style), but what is gained is low-cost alignment across domains and modalities, which is also the prerequisite for LUMIR's decoupling—the representation space no longer depends on the task structure of downstream models.
Interaction-Aware Fusion and Sequential Constraints
Next, an interaction-aware fusion mechanism is designed to model cross-modal interactions. At the same time, sequential signals are introduced as structural constraints to guide the representation learning process. This design ensures that the learned representations both contain multimodal information and preserve the temporal structure required for sequential recommendation.
How contrastive learning and sequential constraints work. The core of contrastive learning is to pull positive sample pairs closer (e.g., representations of the same item under different modalities or different augmented views) and push negative sample pairs apart, thereby shaping discriminative representations without task labels. In multimodal scenarios, this forces information from different modalities to converge in the shared space rather than drown each other out. Sequential constraints address another problem: representations obtained from pure modality alignment may lose the ordering of user behavior, while sequential recommendation relies heavily on "recent preferences" and "transition patterns." Introducing sequential signals as structural constraints is equivalent to adding a temporal regularization layer on top of the contrastive objective, enabling representations to possess both cross-modal consistency and temporal distinguishability. Together, these two aspects determine whether the representations can be directly reused by downstream backbones.
Advantages of Decoupling
Unlike existing methods, LUMIR separates representation learning from the optimization objectives of downstream recommendation models. This means the learned multimodal representations can be trained and reused independently of specific models, enhancing the generality of the representations.
Limitations of the end-to-end tightly coupled paradigm. Traditional multimodal recommendation often embeds fusion modules directly into a recommendation network and trains end-to-end with click or next-visit losses. The representations learned this way are deeply bound to the capacity, inductive bias, and optimization path of a specific backbone; switching to another model requires retraining, and the fusion module is difficult to evaluate independently. Decoupling means treating representation learning as an independent pretraining stage, optimized with self-supervised objectives (such as contrastive learning) to produce a transferable representation; downstream models only perform lightweight adaptation on top of this representation. Its advantages are: representations can be reused across models and scenarios; representation quality can be evaluated independently of specific tasks; and when new backbones are integrated, there is no need to rerun expensive multimodal fusion training, significantly reducing engineering and computational costs.
Experimental Validation: Performance Gains Across Multiple Backbone Models
On a multimodal point-of-interest (POI) dataset under the next-POI recommendation setting, researchers evaluated the representations learned by LUMIR on multiple sequential recommendation backbone models. Results show that LUMIR consistently outperforms existing multimodal fusion baselines on both accuracy-oriented and beyond-accuracy metrics. This demonstrates that decoupling representation learning from downstream optimization is an effective paradigm capable of learning reusable multimodal representations.
The code is publicly available in a GitHub repository (https://github.com/wzehui/LUMIR) for researchers to reproduce and further explore.
Implications for Recommender Systems Research
The proposal of LUMIR reflects a trend in the multimodal recommendation field: moving from end-to-end tightly coupled design toward modular, reusable representation learning. By leveraging LLMs to unify the semantics of heterogeneous modalities and using contrastive learning to decouple representations from tasks, this method provides an approach that subsequent research can draw upon. In the future, how to extend this paradigm to more modalities and recommendation scenarios is worth further exploration.
Related articles

Gemini 4 Neon Released: A Balance Between Frontier Intelligence and Efficient Task Execution
Google DeepMind releases Gemini 4 Neon, focused on frontier intelligence and fast end-to-end task execution. This article breaks down its benchmark performance in coding, enterprise analysis, scientific reasoning, and cybersecurity, its four thinking levels, pricing, and safety alignment data.

Is AI the Culprit Behind Global Outages? The Trust Crisis Behind the NSA and FAA Failures
Fourth major failure in five days hits NSA classified network. Microsoft blames a software bug, FAA denies its new AI system, NSA says routine maintenance. Senator Warner suggests AI accelerates vulnerability discovery, leading to rushed fixes — while 39% of Americans already view AI negatively, exposing a deepening trust crisis.

Robot Developer Builds Custom B-rep Kernel with 3,487 Tests
A robot developer builds a custom B-rep geometry kernel with 3,487 tests, exploring why robotics needs dedicated CAD tools and the barriers to building one.