RaRe
A reasoning-aware relational representation learning framework that uses chain-of-thought rationales generated by multimodal large language models to construct and refine scene graph relation representations, improving generalization to rare and unseen relations in open-vocabulary scene graph generation.
Timeline (last 90 days)
The researchers propose RaRe, a reasoning-aware relational representation learning framework that converts chain-of-thought (CoT) rationales generated by multimodal large language models (MLLMs) into structured relational embeddings and directly optimizes the representation space derived from the rationales.
Through self-supervised contrastive learning, RaRe aligns the CoT rationale embeddings generated by the MLLM with textualized relation descriptions, bringing the rationale representations close to the corresponding relation semantics in the semantic space.
RaRe adopts a specificity-oriented reinforcement learning objective to refine rationale generation, encouraging the model to produce relation evidence that is specific to particular object pairs and semantically more discriminative.
Unlike approaches that treat CoT merely as a reasoning trace, RaRe directly optimizes the representation space derived from the rationales to improve discriminability among relations.
On the VG150 and PSG datasets, RaRe outperforms existing baselines in class-balanced recognition.
The paper notes that the additional gains from learning vary across datasets and proposal settings, and that the effect is related to factors such as data distribution and proposal quality.
The code for RaRe has been made public on the project page.
All Facts (7)
The researchers propose RaRe, a reasoning-aware relational representation learning framework that converts chain-of-thought (CoT) rationales generated by multimodal large language models (MLLMs) into structured relational embeddings and directly optimizes the representation space derived from the rationales.
50%UnverifiedThrough self-supervised contrastive learning, RaRe aligns the CoT rationale embeddings generated by the MLLM with textualized relation descriptions, bringing the rationale representations close to the corresponding relation semantics in the semantic space.
50%UnverifiedRaRe adopts a specificity-oriented reinforcement learning objective to refine rationale generation, encouraging the model to produce relation evidence that is specific to particular object pairs and semantically more discriminative.
50%UnverifiedUnlike approaches that treat CoT merely as a reasoning trace, RaRe directly optimizes the representation space derived from the rationales to improve discriminability among relations.
50%UnverifiedOn the VG150 and PSG datasets, RaRe outperforms existing baselines in class-balanced recognition.
50%UnverifiedThe paper notes that the additional gains from learning vary across datasets and proposal settings, and that the effect is related to factors such as data distribution and proposal quality.
50%UnverifiedThe code for RaRe has been made public on the project page.
50%