Cross-Modal Inconsistency in Omni Models: Separating the Answer Subspace from Divergence Representations

Cross-modal divergence in omni models stems from misaligned modality-specific answer representations, not from the signals that predict it.
Using answer subspace analysis across three omni models and two multimodal tasks, researchers find that information predicting cross-modal inconsistency lies outside the answer subspace in a reproducible 2D representation. Matched interventions show only perturbing the answer subspace changes predictions; perturbing a single modality's answer subspace alone induces divergence, revealing modality misalignment as the cause.
Title
Cross-Modal Inconsistency in Omni Models: Separating the Answer Subspace from Divergence Representations
Meta Description
Omni models often give different answers to semantically equivalent inputs. A new study using answer subspace analysis finds that the representations predicting cross-modal inconsistency are separate from those causing it: divergence information is concentrated in a two-dimensional representation outside the answer subspace, and only perturbing the answer subspace changes predictions. Perturbing the answer subspace of a single modality alone is enough to induce divergence, revealing that misaligned modality alignment is the key.
Article Content
Omni models process audio, vision, and text within a unified language backbone, yet they often give different answers to semantically equivalent inputs. What is the representational mechanism behind this cross-modal inconsistency? A recent study, through answer subspace analysis, reveals a counterintuitive conclusion: the representations that predict divergence are separate from the representations that cause divergence.

Research Background: The Puzzle of Cross-Modal Inconsistency
Omni models map multiple modalities into a shared language representation space, and in theory should produce consistent outputs for equivalent inputs. In practice, however, audio, image, or text inputs with the same meaning often trigger different answers. Existing representation-level analyses fail to distinguish two kinds of information: one that can predict whether divergence will occur, and another whose perturbation actually changes the model's answer. This confusion has kept understanding of the inconsistency mechanism at a superficial level.
The study proposes estimating a low-dimensional answer subspace from answer-position representations, separating answer-related components from the rest of the representation. Across three omni models and two multimodal reasoning tasks, the researchers systematically examined the representational sources of cross-modal inconsistency.
Linear Probing and Representational Decodability
Linear probing is a common tool in mechanistic interpretability research: a simple linear classifier is trained on the activations of a given layer to determine whether some property (e.g., "will the model answer incorrectly," "which modality does the input belong to") can be linearly read out from that layer's representations. If classification accuracy is significantly above chance, the property is said to be "linearly decodable." However, decodability only shows that information exists in the representation and is organized in a roughly linear way; it does not show that the model actually uses this information in downstream computation. Establishing a causal rather than merely correlational relationship requires further intervention experiments (e.g., perturbing activations along specific directions, ablating a subspace) to observe whether the model's output changes accordingly. This study leverages exactly this distinction to separate "representations that can predict divergence" from "representations whose perturbation changes the answer."
Core Finding: Divergence Information Lies Mainly Outside the Answer Subspace
Experiments show that the information predicting cross-modal inconsistency lies mainly outside the answer subspace, concentrated in a reproducible two-dimensional representation that cannot be explained by a fixed modality offset. This means the model contains an internal signal specifically encoding "whether divergence is possible," distinct from the representational channel that actually generates the answer.
Matched intervention experiments further confirm the difference in behavioral roles: perturbing the answer subspace significantly changes model predictions, whereas perturbing the divergence representation (even with the same dimensionality) barely changes predictions, with an effect similar to a random subspace. This contrast clearly shows that linear decodability does not equal behavioral effectiveness—a representation being readable by a linear classifier does not mean perturbing it will change model behavior.
Answer Subspace and Subspace Decomposition
The answer subspace refers to a low-dimensional subspace in the activation space of a given layer (usually at positions corresponding to answer tokens), spanned by the "answer directions" of multiple input–output samples. Researchers extract the directions most relevant to the final prediction by performing principal component analysis or linear discriminant analysis on the answer-position representations of many samples, forming this subspace. Projecting an activation vector onto this subspace yields the "answer-related component," and the remainder is the component "outside the answer subspace." The value of subspace decomposition is that it partitions high-dimensional activations into functionally interpretable pieces, making "where a certain kind of information is located" a testable question. The finding that the signal predicting divergence falls outside the answer subspace was made possible precisely by this decomposition, rather than vaguely stating that "the representation contains divergence information."
The Key to Inducing Divergence: Misalignment of Modality-Specific Answer Representations
The study's most critical finding is that cross-modal divergence can be induced by misaligning modality-specific answer representations. Perturbing the answer subspace of just a single modality is enough to induce divergence, whereas applying the same perturbation to both modalities largely preserves consistency. This suggests divergence arises from the relative offset between answer representations across modalities, not from absolute positions.
This mechanism has direct implications for understanding the robustness of omni models: inconsistency in response to equivalent inputs may not be random noise, but rather the result of imperfect alignment of modality-specific representations. Improving cross-modal consistency may require intervening on modality alignment within the answer subspace, rather than merely adjusting high-level fusion strategies.
Matched Intervention and Causality
Matched intervention is the controlled-variable idea embodied in activation space: applying perturbations of the same dimensionality and similar norm to two conditions while changing only the direction of the perturbation, thereby separating the effect of "perturbation magnitude" from that of "perturbation direction/location." In this study, applying perturbations of equal strength to the answer subspace and to the divergence representation yields a significant change in predictions in the former case and near-none in the latter, ruling out the alternative explanation that "any sufficiently large perturbation will change behavior." Likewise, the different outcomes of applying the same perturbation to a single modality versus to both modalities show that what matters is not the absolute magnitude of the perturbation, but the relative relationship between the answer representations of the two modalities. Matched intervention thus becomes a key experimental design for moving from correlation to causal attribution.
Methodological Implications and Limitations
Through subspace decomposition and matched intervention, the study establishes a causal chain from representation to behavior, providing a reusable framework for analyzing the internal mechanisms of multimodal models. However, the study covers only three models and two reasoning tasks, and the generality of the two-dimensional divergence representation still requires further validation. Moreover, how to translate this finding into practical methods for improving consistency remains an open question.
Key Takeaways
- Predictive information about cross-modal inconsistency is concentrated in a two-dimensional representation outside the answer subspace and is unrelated to modality offset.
- Perturbing the answer subspace changes predictions, while perturbing the divergence representation has almost no effect, proving that linear decodability does not equal behavioral effectiveness.
- Perturbing the answer subspace of a single modality alone can induce divergence, while applying the same perturbation to both modalities preserves consistency.
- Divergence arises from the misalignment of modality-specific answer representations, not from absolute representational differences.
For omni model developers, this study suggests that improving cross-modal consistency requires attention to modality alignment within the answer subspace, rather than merely optimizing the predictive power of shared representations.
One-line Summary
Cross-modal divergence in omni models arises from the misalignment of modality-specific representations within the answer subspace, not from the signal that predicts divergence itself.
Paragraph Summary
This study focuses on the phenomenon in which omni models give different answers to semantically equivalent inputs. It estimates a low-dimensional answer subspace from answer-position representations, separating answer-related components from the rest of the representation. Across three omni models and two multimodal reasoning tasks, the study finds that information predicting cross-modal inconsistency lies mainly outside the answer subspace, concentrated in a reproducible two-dimensional representation that cannot be explained by a fixed modality offset. Matched intervention experiments show that perturbing the answer subspace significantly changes predictions, whereas perturbing the divergence representation has almost no effect, indicating that linear decodability does not equal behavioral effectiveness. Most critically, perturbing the answer subspace of a single modality alone induces divergence, while applying the same perturbation to both modalities preserves consistency, showing that divergence arises from the relative misalignment between modality-specific answer representations.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.