IAU-FOA
A transferable adversarial attack method targeting multimodal large language models. It combines visual invariance augmentation with optimal transport feature alignment, performing fine-grained alignment of adversarial examples at both the global semantic level and the local patch level to improve cross-model transferability in black-box attack scenarios.
Timeline (last 90 days)
IAU-FOA is a transferable adversarial attack method that combines visual invariance augmentation and feature optimal alignment, simultaneously aligning adversarial examples and target samples at both global and local levels.
At the global level, IAU-FOA uses a cosine similarity-based objective function to reduce the global semantic gap between adversarial examples and target samples.
At the local level, IAU-FOA clusters patch tokens into local patterns and performs fine-grained feature matching via optimal transport (OT).
IAU-FOA introduces confidence-adaptive unbalanced transport, relaxing constraints on weakly matched clusters and reducing unreliable local alignment to improve transferability.
IAU-FOA's visual invariance augmentation strategies include bidirectional pixel intensity scaling (simulating exposure and contrast variation) and per-channel white balance adjustment (simulating illumination color temperature variation).
Experiments conducted by the researchers on multiple open-source and closed-source MLLMs show that IAU-FOA consistently outperforms current state-of-the-art methods on targeted transfer attacks.
The code for IAU-FOA has been open-sourced on GitHub (https://github.com/jiaxiaojunQAQ/IAU-FOA).
All Facts (7)
IAU-FOA is a transferable adversarial attack method that combines visual invariance augmentation and feature optimal alignment, simultaneously aligning adversarial examples and target samples at both global and local levels.
50%UnverifiedAt the global level, IAU-FOA uses a cosine similarity-based objective function to reduce the global semantic gap between adversarial examples and target samples.
50%UnverifiedAt the local level, IAU-FOA clusters patch tokens into local patterns and performs fine-grained feature matching via optimal transport (OT).
50%UnverifiedIAU-FOA introduces confidence-adaptive unbalanced transport, relaxing constraints on weakly matched clusters and reducing unreliable local alignment to improve transferability.
50%UnverifiedIAU-FOA's visual invariance augmentation strategies include bidirectional pixel intensity scaling (simulating exposure and contrast variation) and per-channel white balance adjustment (simulating illumination color temperature variation).
50%UnverifiedExperiments conducted by the researchers on multiple open-source and closed-source MLLMs show that IAU-FOA consistently outperforms current state-of-the-art methods on targeted transfer attacks.
50%UnverifiedThe code for IAU-FOA has been open-sourced on GitHub (https://github.com/jiaxiaojunQAQ/IAU-FOA).
50%