Can Multimodal LLMs Both Create and Detect Fake News? New Study Reveals Experiments with 9,000 Fabricated Stories

A Core Question That Remains Unanswered

The rapid advancement of generative AI raises a deeply unsettling prospect: could Multimodal Large Language Models (MLLMs) be weaponized for large-scale disinformation campaigns on social media? Most prior research has focused on text-only misinformation, but a more fundamental question has gone unanswered — can MLLMs be exploited to fabricate convincing multimodal fake news (combining both text and images), and can they reliably detect such content?

This paper published on arXiv tackles exactly that question. The researchers built a multi-agent framework to systematically evaluate the real-world capabilities of today's leading models on both ends of the problem — generating and detecting fake news — with results that are both alarming and foundational for future defenses.

A Multi-Agent Assembly Line for Fake News

The study's core innovation is a multi-agent collaborative framework composed of three specialized agents with distinct roles:

• Story Agent: Generates plausible-sounding fabricated narratives that contradict real news;
• Image Agent: Pairs each fake story with a visually convincing image;
• Critic Agent: Reviews the generated content for quality control, ensuring the fake news is sufficiently believable.

The three agents work in concert, simulating the kind of deliberate, coordinated fabrication seen in real-world disinformation operations. What makes this design particularly clever is that it doesn't merely generate random noise — it purposefully constructs counternarratives designed to refute real news, making the fabricated content more logically coherent and visually deceptive.

Using this framework, the research team generated over 9,000 paired multimodal news posts (matched true/fake pairs) across three domains — science, health, and entertainment — building a substantial evaluation dataset.

A Multi-Agent Framework is a system architecture that decomposes complex tasks across multiple specialized AI agents working in coordination. Each agent handles a specific subtask, collaborating through message passing or shared state to produce results that no single agent could achieve alone. In the context of fake news generation, this division of labor is especially dangerous — it mirrors how real-world disinformation teams operate: copywriters, visual producers, and quality reviewers each playing their part. Compared to a single model generating content directly, a multi-agent pipeline tends to produce more narratively coherent and visually consistent output, because the Critic Agent iteratively filters out logical gaps or obvious visual inconsistencies, retaining only the most deceptive versions.

A Benchmark Test for 16 Models

With this dataset in hand, the researchers ran automated detection benchmarks against 16 open-source and closed-source MLLMs. The goal was straightforward: can these models accurately distinguish real news from fake news fabricated by themselves or similar models?

The results are not encouraging. The study found that most models performed significantly below human-level accuracy. Despite demonstrating strong generative capabilities on the production side, these models fall clearly short on the detection side.

Image Authenticity Is the Critical Weak Point

One of the study's most noteworthy findings is that these models exhibit critical failures in judging image authenticity. In other words, models can often flag suspicious textual narratives, but struggle to identify manipulated or AI-generated images. This weakness exposes a deep structural flaw in current multimodal models' cross-modal consistency verification — they have not yet developed reliable mechanisms for assessing image provenance and authenticity.

This asymmetry — where generating fakes is far easier than detecting them — represents the greatest risk in the fight against disinformation: attackers' capabilities are growing faster than defenders'.

Cross-modal Consistency Verification refers to a model's ability to simultaneously understand text and images, then determine whether the two are semantically and factually aligned. The architecture of current MLLMs typically trains language encoders and visual encoders separately before fusing them, which means models perform reasonably well on single-modality tasks but struggle with deep cross-modal reasoning. For example, a seemingly professional "laboratory photo" paired with fabricated text about a new medical treatment might lead a model to independently judge the text as "broadly plausible" and the image as showing "no obvious signs of AI generation" — while failing to detect the semantic contradictions between the two in specific details. Image authenticity verification faces an additional challenge: the rapid iteration of high-quality image generation models (such as Stable Diffusion and the DALL-E series) has continuously narrowed the visual gap between AI-generated images and real photographs, making detection ever more difficult.

Research Value and Open Resources

The value of this work lies not only in exposing the problem, but in providing the infrastructure needed to build robust social media fake news defense systems. By quantifying the capability gap between generation and detection in current models, the research charts a clear course for developing stronger detection tools — with particular emphasis on improving image authenticity verification.

The research team has also open-sourced the relevant code and dataset (hosted on GitHub), giving the academic and industry communities a reproducible starting point for follow-up research. This open release means more researchers can leverage the 9,000-entry dataset to develop and validate new detection algorithms.

Final Thoughts

This research delivers a clear and urgent message: multimodal LLMs are a far sharper sword than they are a sturdy shield. When AI can fabricate image-rich fake news at low cost and massive scale — while existing models cannot reliably detect it — the risk of information pollution in the social media ecosystem demands serious attention.

For platforms, researchers, and everyday users alike, this is both a warning and a call to action. The real challenge isn't stopping technological progress; it's ensuring detection capabilities can catch up with generative ones — especially by closing the critical gap in image authenticity recognition.