EmoGenie: The First AI-Generated Image Emotion Benchmark — Can MLLMs Truly Read the Feelings in a Picture?

AI image generation keeps getting stronger, but can it really produce the emotion you want? An image labeled "loneliness" might just look like "a cloudy day"; an illustration requested to convey "ecstasy" might be interpreted by the model as "a smile." Even trickier: when Multimodal Large Language Models (MLLMs) evaluate these AI images, do they truly "understand" the emotions within?

A recent study proposes EmoGenie, attempting to systematically answer this question. It is one of the few unified benchmarks that simultaneously covers the emotion expressed by generated images and the emotion evoked in viewers, accompanied by an evaluation model called EmoProbe.

Why do we need a dedicated emotion benchmark?

Most existing affective computing benchmarks are built on natural images — such as facial expressions and scene atmospheres in real photographs. Meanwhile, emotion research on AIGC images is almost entirely concentrated in the portrait domain — checking whether generated faces look happy, sad, or angry.

This leaves an obvious gap:

• What emotions do non-portrait AI images (landscapes, abstract art, object compositions) express?
• What emotions are evoked in viewers when they see these images?
• Are these two levels consistent? Can MLLMs distinguish them?

The paper points out that the emotional effect conveyed by generated images often deviates from the target expectation, and whether MLLMs can reliably interpret the emotions of AI-generated visual content was previously unclear. EmoGenie is designed precisely to fill this gap.

What's inside EmoGenie?

EmoGenie consists of two core components, targeting "question-answering understanding" and "continuous scoring" respectively.

EmoGenie-QA: 90,000 Emotion Questions

The research team used 800 emotion-targeted prompts to drive 8 T2I models to generate 6,400 images, and built question-answer pairs around these images.

EmoGenie-QA includes:

• 90,000 closed-ended questions, covering both single-image and image-pair settings
• 1,057 open-ended questions, requiring models to describe the emotions in images

This means it doesn't just test "choose A or B" — it also examines whether models can articulate emotions clearly in natural language.

EmoGenie-Assess: Continuous Emotion and Aesthetic Ratings

Beyond QA, EmoGenie also collected 13,175 Mean Opinion Scores (MOS), derived from human continuous ratings of images, covering four dimensions:

• Valence: the positive or negative tendency of emotion
• Arousal: the intensity of the emotion
• Dominance: the viewer's sense of control
• Aesthetic quality: how visually appealing the image is

Together, these four dimensions provide a detailed characterization of the emotional experience of AI images, rather than simple "happy/sad" labels.

The Valence-Arousal-Dominance (VAD) model is a widely used three-dimensional framework in affective computing. Valence describes the positive or negative direction of emotion (e.g., happiness is positive, fear is negative), Arousal describes the activation or intensity of emotion (e.g., anger is high-arousal, calm is low-arousal), and Dominance describes whether an individual feels in control or dominated in a situation. Compared to discrete "basic emotion" labels (such as joy, anger, sorrow, fear), VAD captures more nuanced, continuous emotional states and facilitates cross-modal and cross-cultural comparison. Evaluating VAD alongside aesthetic quality means the benchmark cares not only about "what emotion the image conveys" but also about "whether the viewer feels pleasure and finds it visually appealing," thereby getting closer to real aesthetic experience.

EmoProbe: Connecting QA and Regression

To achieve strong performance on this benchmark, the researchers designed EmoProbe. Its approach combines emotion QA with response-aligned, dimension-specific regression heads for continuous score prediction.

Simply put, EmoProbe doesn't just let the MLLM answer questions — it also has it output continuous predictions for dimensions like valence and arousal while answering.

Experimental results reveal two key conclusions:

1. EmoProbe performs better on emotion-related QA tasks;
2. Its emotion and aesthetic predictions correlate better with human ratings.

The paper summarizes this capability as "unified, coarse-to-fine emotion understanding of AI-generated images."

Multimodal Large Language Models (MLLMs) typically consist of a visual encoder and a language model, capable of accepting image and text inputs and generating text responses — for example, GPT-4V, LLaVA, Qwen-VL, etc. They excel at visual question-answering tasks, but their outputs are primarily discrete text, making it difficult to directly produce continuous values. A regression head is a small neural network attached to model features, used to predict continuous scalars such as emotional valence or arousal. EmoProbe combines the two: the MLLM handles understanding and answering emotion questions, while the regression head outputs dimension scores, achieving both "articulating clearly" and "measuring accurately" within a single framework.

What does this mean for AI art and model evaluation?

The value of EmoGenie goes beyond being just a new dataset. It touches on two more practical issues.

First, text-to-image models now have a quantifiable tool for testing "emotional controllability." In the past, we could only say "this image isn't sad enough" based on gut feeling; now we can conduct systematic comparisons using 800 emotion prompts and 6,400 images.

Second, the reliability of MLLMs as "AI judges" needs scrutiny. If MLLMs themselves cannot understand the emotions of AI images, then using them for automatic evaluation, automatic scoring, and automatic filtering of generated results would rest on an unreliable foundation. EmoGenie-Assess uses human MOS as a reference precisely to test this.

For teams working on AI art products, affective computing research, or model evaluation, EmoGenie provides a reusable testbed: it can test both generative models and understanding models.

Limitations and Open Questions

The boundaries of this work are also clear. With 800 prompts, 8 T2I models, and 6,400 images, the scale is not small for an emotion benchmark, but emotion itself is highly subjective, and cross-cultural and cross-individual differences are hard to fully capture with a single MOS.

Additionally, EmoProbe is currently a method proposed alongside this benchmark, and its generalization ability to other image distributions and other MLLMs still requires more independent verification.

One-line Summary

EmoGenie builds an emotion benchmark for AI-generated images and proposes EmoProbe to evaluate MLLMs' emotion understanding capabilities.

Paragraph Summary

This study proposes EmoGenie, an emotion benchmark for AI-generated images that simultaneously covers the emotions expressed by images and those evoked in viewers. The benchmark comprises 800 emotion prompts, 6,400 images generated by 8 T2I models, 90,000 closed-ended QA pairs, 1,057 open-ended questions, and 13,175 human MOS ratings covering valence, arousal, dominance, and aesthetic quality. The accompanying method EmoProbe combines emotion QA with dimension-specific regression heads, outperforming comparison methods in both emotion QA and continuous score prediction. The research provides a reusable testbed for the emotional controllability of text-to-image models and the reliability of MLLMs as AI judges.