MDT Model: Detecting Ambivalence and Hesitancy Through Cross-Modal Conflict Recognition

When Emotions Are "Words Apart": The Core Challenge of A/H Recognition

Human emotional expression is rarely consistent. When a person displays ambivalence or hesitancy during a clinical interview, their facial expressions, vocal tone, and verbal content often send conflicting signals — saying "I'm fine" while their voice trembles and their face betrays uncertainty. This cross-modal disagreement is precisely the key signal for identifying ambivalence and hesitancy (A/H) emotions.

The problem is that the vast majority of multimodal fusion methods are designed to "find consensus" — they integrate information from different channels to arrive at a unified judgment. This approach inherently suppresses disagreements between channels, yet those disagreements are the most important diagnostic signal for A/H emotions. This creates a methodological paradox: standard fusion techniques treat the most valuable information as noise and filter it out.

A study published on arXiv addresses this pain point with a new approach — the Modality Discrepancy Transformer (MDT) — specifically designed to capture, rather than smooth over, cross-modal conflicts.

MDT's Core Idea: From 6-Token to 9-Token

MDT builds on the conflict-aware multimodal fusion framework proposed by Bekhouche et al., with its central innovation being the expansion of the original 6-token design into a 9-token representation.

Three Feature Types, Nine Tokens

These 9 tokens are composed of three parts:

• Three modality embeddings: raw feature representations for the facial, acoustic, and language channels, respectively.
• Three absolute-difference features: explicitly measuring the gap between different modalities.
• Three Hadamard-product discrepancy features: capturing interactive conflicts between modalities through element-wise multiplication.

The latter two feature types are learned through linear projections. The design intent is clear — rather than relying on the model to incidentally learn conflict information, cross-modal discrepancies are explicitly encoded as first-class citizens in the input representation. This is the fundamental distinction between MDT and traditional fusion methods.

The Hadamard product is the element-wise product of two vectors or matrices of the same dimension, denoted as u⊙v, where each position in the result equals the product of the two inputs at that position. Unlike matrix multiplication, the Hadamard product preserves the original dimensional structure and captures the "synergy or opposition" between two vectors at each feature dimension: when two modalities have the same sign on a dimension, the product is positive (synergy); when signs differ, it is negative (opposition).

In MDT's design, absolute-difference features (|u - v|) and Hadamard-product discrepancy features (u⊙v) are complementary: the former measures the "magnitude of distance" between two modality representations — an amplitude-level inconsistency measure; the latter captures their "directional relationship," reflecting finer-grained interactive conflicts. Together they provide richer conflict semantics than a simple difference alone, enabling the Transformer's attention mechanism to understand the nature and intensity of cross-modal inconsistencies from multiple perspectives.

Attention Mechanism for Processing Conflict Signals

These 9 tokens are then processed by the Transformer's self-attention module. The attention mechanism is naturally suited for modeling relationships between tokens, allowing the model to weigh the relative importance of different modality embeddings and discrepancy features — enabling more precise judgments when faced with conflicting signals.

Three Key Architectural Components

MDT's completeness lies not only in its token design but also in three collaborating architectural components.

FiLM Text-Conditioned Modulation

MDT employs a text-conditioned modulation mechanism based on FiLM (Feature-wise Linear Modulation). Linguistic content typically carries the most explicit semantic information in A/H recognition; by having the text signal modulate other features, the model can use semantics as an anchor to interpret the meaning of facial and acoustic signals.

FiLM (Feature-wise Linear Modulation) is a conditional modulation mechanism originally proposed by Perez et al. in 2018 for visual question answering. Its core idea is to use a "conditioning signal" (such as text) to generate a set of scaling parameters γ and shift parameters β, applying a channel-wise affine transformation to another feature map (i.e., FiLM(x) = γ·x + β). This resembles the form of normalization layers like Batch Normalization, but the parameters are dynamically generated by an external signal rather than fixed.

In MDT, the text modality is chosen as the "conditioning source" for good reason: compared to facial expressions and speech, linguistic content typically carries more explicit and structured semantic information. Even when a speaker shows ambivalent emotion, their word choices often still provide reliable context. By using FiLM to let the text signal dynamically modulate the representations of visual and auditory features, the model learns to understand "what does this expression/tone mean in this semantic context," enabling more accurate interpretation of cross-modal conflicts.

LoRA Fine-Tuning

The study uses LoRA (Low-Rank Adaptation) as the core fine-tuning strategy. This choice directly impacts training efficiency — LoRA only updates a small number of parameters, dramatically reducing training costs, which is a key reason MDT can complete training in such a short time.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method proposed by Hu et al. in 2021, designed for adapting large pretrained models to downstream tasks. Its principle is to freeze the vast majority of the original model's weights, injecting two low-rank matrices A and B in parallel alongside the weight matrices to be updated (where the original update ΔW ≈ BA), and training only these small matrices to indirectly modify model behavior. Since the number of trainable parameters is far smaller than full fine-tuning (typically reduced by over 99%), LoRA dramatically lowers memory usage and training time.

This choice is critical to MDT's practical value. Annotated data in A/H recognition is typically scarce (the BAH dataset is limited in scale), and full fine-tuning of large models is not only expensive but also faces serious overfitting risks. LoRA's regularization effect is well-suited for low-data scenarios — it preserves the generalization capability of the pretrained model while allowing targeted adaptation to the specifics of the A/H task. This is the core technical guarantee behind compressing training time to under 20 minutes.

Text-Guided Late Fusion

At inference time, MDT introduces a text-guided late fusion branch that blends a text-only auxiliary prediction head with the full multimodal output. This design serves as a redundant safety net: when multimodal signals are too noisy, the text-only judgment provides a stable fallback.

Experimental Results: 10+ Points Ahead of the Strongest Baseline

MDT was evaluated on the BAH dataset from the 3rd ABAW Challenge, with compelling results:

• Achieved a Macro F1 of 0.7408 on the labeled test set
• Achieved a Macro F1 of 0.7368 on the private leaderboard
• Outperformed the strongest published baseline by over 10 percentage points

Equally noteworthy is the efficiency: the entire training process completes in under 20 minutes on a single GPU. A 10+ point performance gain paired with a sub-20-minute training time demonstrates that MDT achieves its results not by throwing more compute at the problem, but through more principled architectural design — explicitly modeling conflicts — delivering an exceptionally cost-effective breakthrough.

Methodological Implications

The most valuable contribution of MDT may not be the specific F1 numbers, but its critical reflection on the multimodal fusion paradigm. For a long time, fusion has been understood by default as "integrating consistent information." But in scenarios like affective computing and clinical diagnosis, the inconsistency between signals is itself information.

Explicitly encoding absolute differences and Hadamard-product discrepancies as tokens is essentially telling the model: "This contradiction matters — pay close attention to it." This "conflict-aware" approach has broad implications for any multimodal task involving channel disagreement, such as deception detection, mental health assessment, and emotion understanding in human-computer interaction.

Of course, as a preprint, MDT's validation is currently limited to the single BAH dataset, and its generalizability to larger-scale and more diverse clinical scenarios remains to be further examined. But based on existing results, it provides a clear and efficient technical pathway for the automatic recognition of "mixed-signal" emotions like ambivalence and hesitancy.