MDT
A multimodal Transformer model designed to recognize ambivalence and hesitancy emotions. It explicitly encodes cross-modal discrepancy features—absolute difference features and Hadamard product difference features—to capture inter-channel conflict signals, employing a 9-token representation and FiLM text-conditioned modulation.
Timeline (last 90 days)
MDT (Modality Discrepancy Transformer) is a multimodal method for recognizing ambivalent/hesitant (A/H) emotions, specifically designed to capture rather than smooth out cross-modal conflicts.
MDT builds on the conflict-aware multimodal fusion framework proposed by Bekhouche et al., and its core innovation is extending the original 6-token design to a 9-token representation.
MDT's 9 tokens consist of three modality embeddings (facial, speech, language), three absolute difference features, and three Hadamard product difference features.
The absolute difference features and Hadamard product difference features in MDT are learned through linear projection and explicitly encode cross-modal discrepancies as first-class citizens of the input representation.
MDT employs a FiLM (Feature-wise Linear Modulation)-based text-conditioned modulation mechanism that allows text signals to modulate facial and speech features.
MDT uses LoRA (Low-Rank Adaptation) as its primary fine-tuning method to reduce training cost.
MDT introduces a text-guided late-fusion branch at inference time, blending a text-only auxiliary prediction head with the full multimodal output.
MDT achieved a macro-averaged F1 of 0.7408 on the labeled test set of the BAH dataset in the 3rd ABAW Challenge, and a Macro F1 of 0.7368 on the private leaderboard.
MDT outperforms the strongest published baseline method by more than 10 percentage points.
The entire training process of MDT can be completed in under 20 minutes on a single GPU.
1 more timeline events
All Facts (11)
MDT (Modality Discrepancy Transformer) is a multimodal method for recognizing ambivalent/hesitant (A/H) emotions, specifically designed to capture rather than smooth out cross-modal conflicts.
50%UnverifiedMDT builds on the conflict-aware multimodal fusion framework proposed by Bekhouche et al., and its core innovation is extending the original 6-token design to a 9-token representation.
50%UnverifiedMDT's 9 tokens consist of three modality embeddings (facial, speech, language), three absolute difference features, and three Hadamard product difference features.
50%UnverifiedThe absolute difference features and Hadamard product difference features in MDT are learned through linear projection and explicitly encode cross-modal discrepancies as first-class citizens of the input representation.
50%UnverifiedMDT employs a FiLM (Feature-wise Linear Modulation)-based text-conditioned modulation mechanism that allows text signals to modulate facial and speech features.
50%UnverifiedMDT uses LoRA (Low-Rank Adaptation) as its primary fine-tuning method to reduce training cost.
50%UnverifiedMDT introduces a text-guided late-fusion branch at inference time, blending a text-only auxiliary prediction head with the full multimodal output.
50%UnverifiedMDT achieved a macro-averaged F1 of 0.7408 on the labeled test set of the BAH dataset in the 3rd ABAW Challenge, and a Macro F1 of 0.7368 on the private leaderboard.
50%UnverifiedMDT outperforms the strongest published baseline method by more than 10 percentage points.
50%UnverifiedThe entire training process of MDT can be completed in under 20 minutes on a single GPU.
50%UnverifiedMDT's validation is currently limited to the single BAH dataset, and its generalization ability in larger-scale and more diverse clinical settings remains to be examined.
50%