Multimodal Emotion Recognition
A core task in affective computing that integrates text, speech, visual and other modality information to determine a speaker's emotional state, and that faces the challenge of inter-modality conflict.
Timeline (last 90 days)
EmoHarness is an external framework that guides a frozen small model to use multimodal evidence by organizing prompts, settings, and verification code, without updating model weights.
In multimodal emotion understanding, when textual, vocal, and visual cues conflict, small models with limited parameters are prone to misjudgment driven by a single modality.
Under all three source mappings, replacing the original audio-video with unrelated audio-video (RandA/V) reduced accuracy relative to the full input in all 18 comparisons.
The accuracy drops caused by unrelated audio-video replacement in all 18 comparisons remained statistically significant under conversation-cluster inference and BH-FDR correction.
In all six model-dataset pairings, adding the target audio (text+audio) improved emotion recognition accuracy.
For deployers, one cannot assume that all models utilize audio to the same degree; it is worth verifying item by item when selecting models.
The "presence or absence" of multimodal capability is no longer a distinguishing point; whether a model can remain stable under misleading inputs is the focus of the next stage of evaluation.
The performance drop caused by unrelated audio-video replacement (mismatch-specific harm) differs from simple modality absence: the model fuses mismatched audio-video as valid evidence.
Of the six model-dataset pairings, only four retained a significant audio gain after FDR correction.
All Facts (9)
In multimodal emotion understanding, when textual, vocal, and visual cues conflict, small models with limited parameters are prone to misjudgment driven by a single modality.
50%UnverifiedEmoHarness is an external framework that guides a frozen small model to use multimodal evidence by organizing prompts, settings, and verification code, without updating model weights.
50%UnverifiedUnder all three source mappings, replacing the original audio-video with unrelated audio-video (RandA/V) reduced accuracy relative to the full input in all 18 comparisons.
50%UnverifiedThe accuracy drops caused by unrelated audio-video replacement in all 18 comparisons remained statistically significant under conversation-cluster inference and BH-FDR correction.
50%UnverifiedIn all six model-dataset pairings, adding the target audio (text+audio) improved emotion recognition accuracy.
50%UnverifiedFor deployers, one cannot assume that all models utilize audio to the same degree; it is worth verifying item by item when selecting models.
50%UnverifiedThe performance drop caused by unrelated audio-video replacement (mismatch-specific harm) differs from simple modality absence: the model fuses mismatched audio-video as valid evidence.
50%UnverifiedOf the six model-dataset pairings, only four retained a significant audio gain after FDR correction.
50%UnverifiedThe "presence or absence" of multimodal capability is no longer a distinguishing point; whether a model can remain stable under misleading inputs is the focus of the next stage of evaluation.
50%