When Audio-Video Misleads AI: A Robustness Audit of Multimodal Emotion Recognition

Audit shows Omni models can utilize real audio-video, but irrelevant inputs systematically and significantly reduce accuracy.
An audit study of off-the-shelf multimodal emotion recognition (MERC) Omni models isolates modality contribution and risk by fixing textual context while perturbing only audio-video inputs. On Qwen2.5-Omni-7B, Baichuan-Omni-1.5, and MiniCPM-o 2.6, adding target audio improved accuracy in all six model-dataset pairs, with four significant after FDR correction; while replacing original inputs with irrelevant audio-video significantly reduced accuracy in all 18 comparisons. The case study further shows that audio-video information is more likely to be truly utilized when integrated in later stages of the reasoning chain. The research calls for distinguishing modality utilization effectiveness from vulnerability to misleading inputs, and incorporating audio-video substitution stress tests into pre-deployment evaluation.
Title
When Audio-Video Misleads AI: A Robustness Audit of Multimodal Emotion Recognition
Meta Description
A multimodal emotion recognition audit finds: adding real audio-video to Omni models improves accuracy, but substituting irrelevant audio-video significantly degrades accuracy across all 18 comparisons, revealing model vulnerability to misleading inputs.
Article Content
Multimodal Emotion Recognition (MERC) models can process text, speech, vision, and conversational context simultaneously—but "being able to receive audio-video" does not equal "being able to use it reliably." An audit study of off-the-shelf Omni models reveals an unsettling reality: irrelevant audio-video inputs systematically drag down model accuracy, and this damage is highly statistically significant.

Audit Design: Fix Text, Perturb Audio-Video
The researchers' approach was clever—keeping the target transcript text and prior textual context unchanged, while only adding, removing, or replacing the target audio-video. This isolates the impact of audio-video from textual information, examining its contribution and risk separately.
The main evaluation covered three models—Qwen2.5-Omni-7B, Baichuan-Omni-1.5, and MiniCPM-o 2.6—on the MELD dataset and the full five-session six-class IEMOCAP. There were three core comparisons: text+audio (T+A) versus text-only, full audio-video (Full) versus irrelevant audio-video (RandA/V), and modality-omission baselines.
FDR Correction and BH-FDR: FDR (False Discovery Rate) correction is a method for correcting multiple hypothesis tests. When a study conducts multiple statistical tests simultaneously (e.g., across multiple model-dataset pairs), the false positive rate accumulates. BH-FDR (Benjamini-Hochberg method) balances discovery power and false positive risk by controlling the proportion of falsely rejected null hypotheses. "Significant after FDR correction" in the text means results remain statistically reliable after accounting for multiple comparisons, not merely significant in a single test.
Conversational Cluster Inference: Since both MELD and IEMOCAP are conversational data, samples within the same conversation are highly correlated (emotional expression is influenced by context). Cluster-robust inference treats samples from the same conversation as a "cluster" and accounts for within-cluster correlation when computing standard errors, avoiding overestimation of statistical significance due to non-independent samples. This is a key practice for ensuring valid inference in conversation-level emotion analysis.
Real Modalities Provide Gains, but Statistical Strength Varies
Across all six model-dataset pairs, adding target audio (T+A) improved accuracy, with four pairs remaining significant after FDR correction. This shows that audio-video is not mere decoration—in most cases it does carry emotional cues beyond text.
However, only four of six pairs were significant, meaning the robustness of gains varies. For deployers, this is a reminder: you cannot assume all models utilize audio to the same degree, and it's worth verifying item by item during model selection.
Meaning of T+A vs. Full Comparison: The T+A (text+audio) versus text-only comparison measures the incremental contribution of audio; the Full (text+audio+vision) versus T+A comparison further isolates the role of the visual modality. The "modality-omission baseline" refers to performance after successively removing a modality, used to determine the marginal value of each modality. This design can distinguish "multimodal overall effectiveness" from "individual modality effectiveness," providing direct guidance for model selection and fusion strategies.
Misleading Inputs Cause Widespread and Significant Damage
More noteworthy are the negative results. Under three source mappings, replacing original audio-video with irrelevant audio-video (RandA/V) reduced accuracy in all 18 comparisons relative to full input, and all 18 remained significant under conversational cluster inference and BH-FDR correction—whether counting only jointly valid pairs or treating invalid/insufficient-VRAM results as errors, the conclusion holds.
This overwhelming "18/18" result shows that multimodal models' sensitivity to audio-video channels is a double-edged sword: real signals bring gains, false signals bring harm. Comparison with modality-omission baselines further distinguishes two effects—substitution penalty differs from simple absence; it points to "mismatch-specific harm," where models treat mismatched audio-video as valid evidence.
Mismatch-Specific Harm: This refers to models not simply ignoring irrelevant inputs, but erroneously fusing mismatched audio-video as valid evidence consistent with textual emotion, producing more severe performance degradation than missing modalities. This harm is specific—it arises from cross-modal semantic inconsistency rather than reduced information. This suggests fusion mechanisms lack validation of inter-modality consistency, a common weakness of current Omni models.
Case Study: When Information Is Truly Utilized
The Qwen/MELD case study provides mechanistic clues. In decision-query audio knockout experiments, relative to edge-matched Video-Sham controls, gold-vs-best-candidate marginal recovery at the Late stage was significantly greater than at Early or Middle (Holm-corrected test). This suggests that when audio-video information is integrated in later stages of the reasoning chain, models are more likely to truly utilize it rather than superficially receive it.

Decision-Query Audio Knockout Experiment: A causal attribution method. By "knocking out" (removing or replacing) specific audio segments, one observes changes in the model's marginal scores for target candidates (gold vs. best-candidate), thereby determining whether that audio is actually used in decision-making. Late/Early/Middle refer to knockout occurring at different stages of the reasoning chain (e.g., shallow fusion vs. deep fusion). If Late-stage knockout causes significantly greater marginal recovery, it indicates the model truly integrates audio information in later reasoning stages rather than superficially receiving it.
Holm-Corrected Test: A multiple comparison correction method controlling the family-wise error rate (FWER), more conservative than Bonferroni but with greater statistical power. Used in the text to compare differences across Early, Middle, and Late stages, ensuring conclusions remain robust under multiple comparisons.
Implications for Practitioners
The value of this audit lies in distinguishing "modality utilization effectiveness" from "vulnerability to misleading inputs." The former asks whether models use audio-video; the latter asks what happens when they use it wrongly. The answer to both questions can simultaneously be "yes"—which is precisely the situation of current Omni models.
For teams building conversational emotion analysis systems, there are several directly actionable points: conduct audio-video substitution stress tests before real deployment, not just ablation experiments; pay attention to performance under boundary conditions such as invalid/insufficient VRAM, as they may be counted as errors; and carefully choose the timing and level of audio-video fusion—the case study suggests later-stage integration may be more effective.
The "presence or absence" of multimodal capabilities is no longer the differentiator; whether models can remain stable under misleading inputs is the evaluation focus of the next stage.
One-line Summary
Audit shows Omni models can utilize real audio-video, but irrelevant inputs systematically and significantly reduce accuracy.
Paragraph Summary
An audit study of off-the-shelf multimodal emotion recognition (MERC) Omni models isolates modality contribution and risk by fixing textual context while perturbing only audio-video inputs. On Qwen2.5-Omni-7B, Baichuan-Omni-1.5, and MiniCPM-o 2.6, adding target audio improved accuracy in all six model-dataset pairs, with four significant after FDR correction; while replacing original inputs with irrelevant audio-video significantly reduced accuracy in all 18 comparisons. The case study further shows that audio-video information is more likely to be truly utilized when integrated in later stages of the reasoning chain. The research calls for distinguishing modality utilization effectiveness from vulnerability to misleading inputs, and incorporating audio-video substitution stress tests into pre-deployment evaluation.
Related articles

Gemini 4 Hydrogen vs Sonnet Comparison Test: Uses Only 1% of Weekly Quota, Performance on Par
Hands-on test: Gemini 4 Hydrogen vs Sonnet on the same CRUD task—Hydrogen used only 1% weekly quota with comparable performance and better UI details, while Anthropic's model used 7%. Developer says daily tasks don't require expensive models.

Gemini 4 Neon Released: Beats Opus 5.5 on Multiple Benchmarks, Google Returns to the Top Tier
Google DeepMind releases Gemini 4 Neon: 61 on the Artificial Analysis intelligence index (3rd) and 1st on the coding index, beating Opus 5.5 on multiple benchmarks. $3/$15 per million tokens; available in the Gemini API and AI Studio.

Gemini 4 Neon Review: Benchmark Scores Rival Carbon, API Cost-Performance Analysis
Full review of Google Gemini 4 Neon: benchmark scores rival Carbon, API priced at 2x Hydrogen, yet Antigravity quota consumption is only 1x (vs Carbon's 3x). Real tests show it replaces 90% of Carbon use cases. Includes channel restrictions, ban risks, and combination strategy analysis.