LUMIR
A contrastive-learning-based framework for decoupled multimodal representation learning. It uses an LLM to map heterogeneous modalities into a shared text space and combines interaction-aware fusion with sequential constraints, decoupling representation learning from downstream recommendation model optimization to improve the generality and reusability of the representations.
Timeline (last 90 days)
LUMIR is a decoupled multimodal representation learning framework based on contrastive learning.
LUMIR uses a large language model (LLM) to map heterogeneous modality information such as images and text into a shared textual representation space.
The cost of using the text space as the shared representation space is that information may be compressed or lost during transcription (e.g., fine-grained visual style).
LUMIR designs an interaction-aware fusion mechanism to model cross-modal interactions and introduces sequential signals into the representation learning process as structural constraints.
LUMIR separates representation learning from the optimization objective of the downstream recommendation model, so the learned representations can be trained and reused independently of a specific model.
Under the decoupled paradigm, when a new backbone is onboarded there is no need to rerun expensive multimodal fusion training, which can significantly reduce engineering and compute costs.
On multimodal point-of-interest (POI) datasets, with next-POI recommendation as the setting, the researchers evaluated the representations learned by LUMIR on multiple sequential recommendation backbones.
Experimental results show that LUMIR consistently outperforms existing multimodal fusion baselines on both accuracy-oriented metrics and beyond-accuracy metrics.
The LUMIR code has been publicly released in the GitHub repository https://github.com/wzehui/LUMIR.
All Facts (9)
LUMIR is a decoupled multimodal representation learning framework based on contrastive learning.
50%UnverifiedLUMIR uses a large language model (LLM) to map heterogeneous modality information such as images and text into a shared textual representation space.
50%UnverifiedThe cost of using the text space as the shared representation space is that information may be compressed or lost during transcription (e.g., fine-grained visual style).
50%UnverifiedLUMIR designs an interaction-aware fusion mechanism to model cross-modal interactions and introduces sequential signals into the representation learning process as structural constraints.
50%UnverifiedLUMIR separates representation learning from the optimization objective of the downstream recommendation model, so the learned representations can be trained and reused independently of a specific model.
50%UnverifiedUnder the decoupled paradigm, when a new backbone is onboarded there is no need to rerun expensive multimodal fusion training, which can significantly reduce engineering and compute costs.
50%UnverifiedOn multimodal point-of-interest (POI) datasets, with next-POI recommendation as the setting, the researchers evaluated the representations learned by LUMIR on multiple sequential recommendation backbones.
50%UnverifiedExperimental results show that LUMIR consistently outperforms existing multimodal fusion baselines on both accuracy-oriented metrics and beyond-accuracy metrics.
50%UnverifiedThe LUMIR code has been publicly released in the GitHub repository https://github.com/wzehui/LUMIR.
50%