Evidence Order Can Change Model Judgments: The "Evidence Noncommutativity" of Multimodal LLMs

A Neglected Variable: Where Evidence Appears

When multimodal large language models (MLLMs) process combinations of images, speech, and text, they frequently face a thorny problem: when image or audio content contradicts the accompanying text, which source does the model trust more? Past research has typically measured this modal preference using "text reliance," but a newly published arXiv paper (arXiv:2609.26986v1) argues that such measurements may contain a long-underestimated confounding factor — the order in which evidence appears.

The authors' core observation is that prior studies on "text bias" tended to fix the arrangement of evidence, or moved task instructions together with the evidence, making it impossible to isolate the contribution of order itself. In other words, what we thought we were measuring — a model's modal preference — may have inadvertently included a positional effect all along.

MLLMs are systems built on top of pretrained language models, capable of processing multiple input modalities including text, images, and audio. Prominent examples include GPT-4V, Gemini, and LLaVA. These models typically map image or audio features into the same embedding space as text via a projection layer or adapter, then pass everything to a Transformer decoder for unified processing. This architecture means that information from different modalities is concatenated into a single continuous context sequence before entering the attention mechanism — image tokens, audio tokens, and text tokens each occupy distinct positions in the sequence. That "position" is not merely logical ordering; in the Transformer's attention computation, it corresponds to different positional encodings. Whether a token appears early or late in the sequence fundamentally affects how much it contributes to the final generated output — and this is the underlying mechanism that makes the positional effect studied in this paper possible.

Isolating "Order" as a Variable with Paired Comparisons

To cleanly quantify the effect of position, the paper adopts a paired comparison experimental design. The elegance of this design lies in its strict control of variables: task instructions remain fixed, the content of both evidence sources (e.g., an image and a piece of text) remains fixed, and the only thing that changes is the order in which these two sources appear.

This approach — swapping only the position while holding everything else constant — completely disentangles two factors that previous research had conflated: modal preference and evidence position. Any change in the model's answer after swapping the order can be directly attributed to position itself, rather than being contaminated by differences in content or instructions. This is the most critical methodological contribution of the paper.

The paired comparison design is a classic method in experimental psychology and causal inference. Its core idea is to eliminate individual-level noise by having the same subject experience both conditions. In machine learning evaluation, this design is particularly effective: the same model instance receives two versions of input — "image first, text second" and "text first, image second" — with identical content. Any difference in output can only be attributed to the ordering. This is equivalent to the logic of a paired t-test in statistics — through self-comparison, statistical power is substantially increased, enabling statistically significant conclusions even from relatively small experiment sets. The paper uses this design to surgically separate the order effect from modal preference.

Key Finding: Placing Perceptual Evidence Last Makes It More "Persuasive"

Experiments spanning both vision and speech models converge on a consistent conclusion: when an image or audio recording is placed after conflicting text, the model's answer shifts noticeably toward the image or audio content.

In other words, given the same contradictory set of evidence, the model is more likely to trust the perceptual evidence simply because it appears later. This reveals what the authors call cross-modal evidence noncommutativity:

• The same evidence, presented in different orders, can lead to different judgments;
• Placing perceptual evidence last increases the model's reliance on its content.

The term "noncommutativity," borrowed from mathematics, is remarkably apt — just as matrix multiplication satisfies AB ≠ BA, a conclusion drawn with evidence A first and B second is not necessarily the same as one drawn with B first and A second. The order of information becomes a substantive factor in shaping the model's judgment.

From the perspective of the Transformer attention mechanism, "later content having more influence" is not counterintuitive. On one hand, autoregressive generation models have a natural "recency preference" for recent context (i.e., the tail of the sequence) when predicting the next token — a phenomenon well-documented in text-only research as "recency bias." On the other hand, during instruction fine-tuning of many MLLMs, task descriptions typically follow immediately after the evidence, so models may have learned the inductive bias of "prioritizing the most recently read evidence before answering." Cross-modal evidence noncommutativity likely reflects the compounded effect of both mechanisms in a multimodal setting, rather than any "subjective preference" the model has for image or audio content per se.

Revisiting Old Conclusions: How Experimental Design Can Mislead Us

The paper does not stop at introducing a new phenomenon — it also revisits prior related work and identifies why those experimental designs may have led to misleading conclusions.

If earlier studies measuring "text bias" consistently placed text first and images second (or vice versa), then the modal preference they observed may have been partly or even largely attributable to a positional effect rather than any genuine preference for a particular modality. This means widely circulated conclusions like "MLLMs trust text more" may need to be re-examined once the order variable is properly controlled.

This carries an important warning for the entire multimodal evaluation field: when designing conflict scenarios to probe model behavior, the arrangement of evidence is not an inconsequential detail that can be arbitrarily fixed — it is an experimental variable that must be explicitly controlled.

Implications for Practice and Evaluation

For researchers, this work serves as a reminder that modal-conflict experiments must include order-controlled comparisons; without them, conclusions about "modal preference" may not hold up. For application developers, the finding also has direct engineering implications: if perceptual evidence (images, audio) and textual descriptions in a multimodal prompt might conflict, their relative position in the context will concretely affect which source the model ultimately trusts.

Looked at from another angle, the positional effect also offers a potential "steering" mechanism — if you want the model to weight image content more heavily, placing the image after the text may be a low-cost strategy worth trying. That said, the paper itself focuses on revealing the phenomenon and correcting evaluation methodology; robustness in real-world applications remains to be further validated.

Conclusion

The value of this paper lies not in proposing a complex new model, but in using a clean experimental design to expose a long-hidden methodological blind spot in multimodal evaluation. When the same evidence can lead to different judgments simply because of ordering, many of our existing assumptions about model "preferences" may need to be questioned. Evidence noncommutativity deserves a place in every future multimodal reliability evaluation.