Multimodal Large Language Model

Image-to-Text leverages the visual understanding capabilities of multimodal large language models to map visual features to natural language descriptions.

一张典型手机照片在多模态大语言模型中约消耗1000-1500个token

Computer Use Agent的核心技术依赖于多模态大语言模型的视觉理解能力，通过截图-分析-执行的循环机制工作

The vast majority of multimodal large language models (MLLMs) are still optimized around single-image input, and with multi-image input the visual token encoding overhead grows linearly or even superlinearly with the number of images.

WOVEN's argument essentially attributes part of MLLMs' deficiencies to a lack of world model capability—that is, although the model can describe image content, it cannot simulate how it evolves with actions.

Multimodal large language models (MLLMs) refer to large-scale models capable of processing multiple modal inputs such as text, images, and video simultaneously, such as GPT-4V and Qwen-VL.

Current multimodal large language models often fail to recognize text in low-resolution, blurry embedded-text scenarios, and even report that there is no text in the image.

Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box attack scenarios.

A new arXiv paper selects medical image alignment evaluation as a scenario for testing the general visual reasoning capabilities of frontier multimodal large language models.

The architecture of current MLLMs typically trains language encoders and visual encoders separately before fusing them, causing models to struggle with deep cross-modal reasoning.

The tested models exhibited critical failures in judging image authenticity, representing a fatal weakness in their detection capability.

Research shows that multimodal large models surpass their ability to detect fake news in their ability to generate fake news, meaning attacker capabilities are growing faster than defender capabilities.

The research constructed a multi-agent collaborative framework composed of three agents with distinct roles — a story agent, an image agent, and a review agent — for generating multimodal fake news.

The researchers conducted benchmark testing of automated fake news detection capabilities on 16 open-source and closed-source MLLMs.

The study found that the fake news detection accuracy of most tested models was significantly lower than human-level performance.

An arXiv paper (arXiv:2609.26986v1) points out that the sequential order in which evidence appears is a long-underestimated confounding factor when measuring the modality preferences of multimodal large models.

Multimodal large language models typically map image/speech features into the same embedding space as text via projection layers or adapters, which are then processed uniformly by the Transformer decoder.

Experiments spanning vision models and speech models yield a consistent conclusion: when an image or audio recording is placed after conflicting text, the model's answer shifts noticeably toward the image/speech content.

In the instruction fine-tuning phase of many MLLMs, task descriptions often immediately follow the evidence, which may cause the model to learn an inductive bias of prioritizing the most recently read evidence before responding.

The so-called modality preference observed in previous research may be partly or largely attributable to positional effects, and the conclusion that "MLLMs trust text more" may need to be re-examined after controlling for the order variable.