IAU-FOA: A New Transferable Adversarial Attack Method Against Closed-Source Multimodal Large Models

The Security Blind Spot of Closed-Source Multimodal Large Models

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in image-text understanding, visual question answering, and related tasks — but they are far from invulnerable. Research shows that these models remain susceptible to transferable adversarial examples, especially in black-box attack scenarios where attackers have no access to the target model's internal architecture and must rely on publicly available open-source surrogate models to craft attacks.

This raises a critical security concern: attackers can use publicly accessible open-source models to generate adversarial examples that "transfer" to commercial closed-source models. As more closed-source MLLMs are deployed in real-world products, the practical threat posed by such attacks deserves serious attention across the industry.

Limitations of Existing Attack Methods

Current mainstream targeted transfer attacks primarily rely on global image-level features for alignment — for example, the [CLS] embedding vector from an encoder. The idea is to bring the adversarial example semantically closer to the target sample at a coarse, holistic level.

However, the paper argues that this coarse-grained alignment has a significant weakness: it fails to leverage the patch-level visual structure (local block-level information) within images. When attacks need to transfer across heterogeneous closed-source MLLMs with vastly different architectures, relying solely on global feature alignment severely limits transferability.

In other words, different visual encoders process local image details in very different ways. Focusing only on global semantics makes it difficult for adversarial examples to generalize across diverse model architectures.

---

The [CLS] embedding vector is a special token (Classification Token) in the Transformer architecture, originally introduced by BERT. In Vision Transformers (ViT), an image is split into fixed-size patches (e.g., 16×16 pixels), each mapped to a token, while the [CLS] token is prepended to the sequence. After passing through multiple self-attention layers, it aggregates global semantic information from the entire image and is typically used as the image-level global representation for downstream tasks. Targeting the [CLS] embedding for attacks is therefore a natural choice — one simply needs to align the adversarial sample's global representation with that of the target. However, the aggregation process of the [CLS] token discards a large amount of spatial and local information, and different encoder architectures encode local textures, edges, and fine details in substantially different ways — making attacks that only align [CLS] embeddings limited in cross-architecture generalization.

---

The Core Design of IAU-FOA

To address these limitations, the researchers propose IAU-FOA (Visual-Invariance-Augmented Feature Optimal Alignment with Adaptive Unbalanced Transport) — an attack method that combines visual invariance augmentation with feature-level optimal alignment. Its core idea is to align adversarial and target samples simultaneously at both global and local levels.

Dual-Level Global and Local Alignment

At the global level, IAU-FOA uses a cosine similarity-based objective to minimize the global semantic gap between the adversarial example and the target sample.

At the local level, the method clusters patch tokens into compact local patterns and performs fine-grained feature matching using Optimal Transport (OT). This step allows the attack to capture the local visual structure within images, compensating for the shortcomings of methods that only consider global features.

---

Optimal Transport (OT) is a mathematical framework for measuring the "transportation cost" between two probability distributions, originating from the earth-moving problem posed by Monge in the 19th century and formalized in its modern form by Kantorovich. Intuitively, OT answers the question: "How do you move a pile of 'sand' (source distribution) into another shape (target distribution) at the lowest total cost?" In machine learning, OT is commonly used to measure distances between feature distributions (i.e., the Wasserstein distance), and is widely applied in generative models, domain adaptation, and similar tasks, thanks to its ability to capture geometric structure rather than merely comparing histograms. In IAU-FOA, patch tokens from the image are clustered and treated as discrete probability distributions. By solving the OT problem, the method finds the optimal matching between local clusters of the adversarial sample and those of the target, enabling fine-grained alignment at the local structural level — rather than simply computing mean feature distances.

---

Confidence-Adaptive Unbalanced Transport

Traditional Balanced OT enforces fixed marginal mass constraints on all local clusters, even when some clusters have no reliable correspondence in the target sample. This forced matching introduces misleading alignment gradients that can actually degrade attack performance.

To address this, IAU-FOA introduces confidence-adaptive unbalanced transport, which relaxes constraints on weakly matched clusters. For local clusters that cannot find reliable correspondences, forced matching is no longer imposed, thereby reducing unreliable local alignments and improving the transferability of adversarial examples.

---

Balanced OT requires strict conservation of marginal mass between source and target distributions — every source cluster's "mass" must be fully allocated, and every target cluster's "mass" must be fully received. This works well when the two distributions have similar structures, but when some local clusters cannot find reasonable correspondences in the target (e.g., the adversarial sample contains certain textures that are entirely absent from the target image), Balanced OT will forcibly assign them to the nearest but actually mismatched target cluster, producing erroneous gradient signals. Unbalanced OT addresses this by incorporating penalty terms such as KL divergence into the objective, allowing marginal constraints to be satisfied "softly." This means low-confidence matches contribute smaller weights, preventing noisy gradients from forced alignment from interfering with the overall optimization direction.

---

Visual Invariance Augmentation Strategy

Another major contribution of the paper is an investigation into how input transformations affect attack transferability, leading to the proposal of visual-invariance augmentation.

This strategy simulates real-world visual variations through two mechanisms:

• Bidirectional pixel-intensity rescaling: simulates changes in exposure and contrast;
• Per-channel white-balance adjustment: simulates changes in lighting and color temperature.

By incorporating these transformations during adversarial perturbation optimization, IAU-FOA encourages the generated perturbations to generalize across different visual encoders. Regardless of the image preprocessing or encoding method used by the target model, the adversarial examples remain effective — which is precisely the key to successful transfer attacks.

Experimental Results and Industry Implications

The researchers conducted extensive experiments on multiple open-source and closed-source MLLMs, with results showing that IAU-FOA consistently outperforms current state-of-the-art methods in targeted transfer attacks. The related code has been open-sourced on GitHub (https://github.com/jiaxiaojunQAQ/IAU-FOA).

From a security research perspective, this work carries dual significance. On one hand, it reveals the real-world attack risks faced by closed-source commercial MLLMs, reminding model providers that being "closed-source" is no guarantee of safety. On the other hand, the ideas behind fine-grained local alignment and visual invariance augmentation provide a clear attack surface for future defense research — understanding how attacks exploit local features and visual variations is essential for building more robust multimodal models.

For practitioners in AI security and adversarial example research, the optimal transport modeling and unbalanced constraint relaxation mechanisms in IAU-FOA offer a compelling technical paradigm worth learning from.