Human vs. Multimodal LLM Differences in Associative Creativity: An Abstract Visual Stimulus Experiment

Research Background: When Visual Stimuli Are Hard to Name

In human communication, conceptual association is a core ability. But when we face visual stimuli that are difficult to map onto familiar semantic concepts, how do humans and multimodal large language models (MLLMs) adjust their associative strategies? A recent study systematically compared the associative creativity of both sides by controlling the naming difficulty of visual stimuli.

The study used two datasets: SilCC modulates naming difficulty through progressively complex combinations of abstract shapes; PixCC increases recognition difficulty by adding random visual noise. Participants included humans and several advanced MLLMs, and evaluation focused on two core metrics of creativity—originality and effectiveness.

Experimental Design: Dual Datasets and Difficulty Manipulation

SilCC: The Naming Challenge of Abstract Shape Combinations

The SilCC dataset progressively increases the complexity of semantic mapping by combining multiple abstract shapes. For example, it moves from simple geometric figures to multi-layered nested non-nameable shapes, forcing participants to abandon direct naming and instead produce associative descriptions. This design simulates real-world visual experiences that are "hard to put into words."

PixCC: Visual Recognition Under Noise Interference

PixCC, by contrast, progressively reduces visual clarity by overlaying random noise on images. As the noise level increases, images gradually lose recognizable semantic features, and participants must extract cues from low-level visual attributes (such as color and texture) rather than relying on overall semantics.

Low-level visual attributes refer to basic features in an image that do not depend on semantic recognition, including color, brightness, contrast, texture, edges, and spatial frequency. In contrast, holistic semantic association relies on object recognition and conceptual categorization, for example, judging a blurry shape as "looks like an animal." Under noise interference, high-level semantic features are the first to be damaged, while low-level features can still be extracted. Humans can actively shift attention from "what is this" to "what texture/color/shape does it look like," and this downward shift in strategy from the semantic level to the perceptual level is an important manifestation of cognitive flexibility in coping with visual uncertainty.

Key Findings: Human Associations Are More Diverse

Higher Semantic Distance and Entropy

Compared with MLLMs, the associative descriptions produced by humans showed higher diversity across multiple dimensions: greater semantic distance (the degree to which descriptions deviate from reference vocabulary), higher entropy (the disorder of the description distribution), and more novel expressions. This means humans can break out of conventional semantic frameworks and generate more unique associations.

Strategy Divergence as Difficulty Increases

In SilCC, as naming difficulty rose, the effectiveness of all participants declined. But in PixCC, humans showed a clear strategic shift: from holistic associations (such as "looks like an animal") to describing low-level visual attributes (such as "a gray block with blurred edges"). MLLMs, however, were more inclined to persist with semantic-level associations even when visual information was already severely insufficient.

Semantic distance is one of the key quantitative indicators for measuring associative creativity, usually defined as the degree of deviation between generated descriptions and a reference semantic space (such as word-vector space or a concept ontology). The greater the distance, the more the description departs from convention and the more novel it is. Entropy is borrowed from information theory and is used to characterize the disorder of a set of descriptions across semantic categories: the higher the entropy, the more dispersed and less concentrated the participants' associations are on a few common concepts, thus reflecting greater diversity. Combined, the two can distinguish "random nonsense" from "meaningful originality": the former has large semantic distance but low effectiveness, while the latter achieves high distance and high entropy while remaining understandable.

Implications for Multimodal Model Development

This study reveals the limitations of current MLLMs in associative creativity: they tend to rely on semantic patterns in training data rather than flexibly adapting to visual uncertainty. For tasks requiring high creativity (such as art generation and metaphor understanding), models may need to introduce more flexible associative mechanisms, for example, dynamically adjusting the weights of semantic and perceptual information.

At the same time, humans' strategic shift under difficult conditions (from semantics to perception) offers a biomimetic idea for model design: when semantic mapping fails, turning to low-level feature description may be an effective adaptive strategy.

Multimodal large language models (MLLMs) are typically composed of a vision encoder and a large language model connected through an alignment layer, and their training objective focuses on mapping visual inputs to semantic concepts already mastered by the pretrained language model. This architecture enables models to perform excellently on standard recognition tasks, but when visual input is blurry or lacks corresponding naming concepts, they easily "force" semantic priors from training data rather than falling back to low-level perceptual description. The so-called dynamic adjustment of the weights of semantic and perceptual information means adaptively adjusting the contribution ratio of semantic priors and low-level features at inference time according to visual uncertainty, and it is a potential mechanism for improving models' associative creativity.

Conclusion

Humans remain significantly superior to multimodal large language models in associative creativity, especially when facing visual stimuli that are hard to name, where humans can produce more diverse and more original descriptions. As difficulty increases, humans flexibly adjust the level of association, while models show a certain rigidity. This finding provides direction for improving the creativity of MLLMs.

One-line Summary

Research reveals that under hard-to-name visual stimuli, human associations are more diverse and flexible than those of multimodal large models.

Paragraph Summary

This study manipulated the naming difficulty of visual stimuli through two datasets, SilCC (abstract shape combinations) and PixCC (random noise interference), and systematically compared the associative creativity of humans and multimodal large models, using originality and effectiveness as core metrics. Results show that human-produced associative descriptions have greater semantic distance, higher entropy, and more novel expressions; as difficulty rises, humans under noise conditions shift from holistic semantic association to low-level visual attribute description, while MLLMs tend to persist with semantic-level association. The study points out that MLLMs over-rely on semantic patterns in training data and lack flexible adaptation to visual uncertainty, and it proposes a biomimetic idea for model design: dynamically adjusting the weights of semantics and perception.