Three Core Gaps in Multimodal LLMs: Frontier Research from Microsoft Research India

Introduction: Multimodal LLMs Still Lag Behind Humans

At a workshop on multimodal and embodied intelligence hosted by Microsoft Research India (MSR India), researchers from MSR India, IIIT Delhi, IIIT Hyderabad, IIT Jodhpur, and other institutions shared their in-depth explorations of multimodal intelligence. A central observation that ran throughout the event: while text-based LLMs have already surpassed humans on a wide range of text tasks, multimodal LLMs still fall significantly short of human performance on visual tasks.

Even more concerning is a structural issue: virtually all multimodal LLMs today are essentially built by "adapting" non-text modalities onto a pure text LLM — plugging in visual, audio, and other information via external encoders. The dominant implementation approach uses a pretrained text LLM as the backbone, converts images into token sequences via a visual encoder (typically CLIP or a ViT variant), and aligns them with the text space through a linear projection layer or cross-attention mechanism. The advantage of this architecture is that it reuses powerful text language model capabilities at relatively manageable training costs — but the structural cost is significant: visual information is compressed and re-encoded before entering the language model, and raw pixel-level details, spatial relationships, and temporal structure are often heavily lost in the process. Vineet from MSR India stated plainly in his opening talk that this raises a fundamental question: Is this "text-centric" architecture truly the right path for handling multimodal intelligence?

As enterprise communications, healthcare, and the rise of physical and embodied AI drive rapid growth in non-text data, if model architectures continue treating vision as an "appendage to text," the ceiling on multimodal capabilities may prove very hard to break through.

The First Gap: Perception — Models That "Cannot See"

Tanuja's team at MSR India started by examining visual reasoning datasets and quickly discovered that existing models — both closed-source and open-source — performed poorly on IQ-test-style benchmarks, and the failures often originated at the perception level. Yet the field at the time lacked diagnostic benchmarks specifically designed to isolate visual perception capabilities.

To address this, the team built a benchmark called "Do You See Me?", inspired by human perceptual ability tests (such as visual tasks used to identify conditions like ADHD). These tasks require no domain knowledge — for example: "How many triangles are in the image?", "How many red triangles?", and visual shape constancy judgments. Shape constancy is one of the foundational capabilities of the human perceptual system — the ability to recognize the same object's shape even under different viewpoints, lighting conditions, or occlusions. Humans develop this ability early through embodied experience, and current multimodal models lack an equivalent learning mechanism, leaving them at an inherent disadvantage on such tasks.

The test results were highly revealing:

• Ordinary humans achieve over 90% accuracy on these simple perceptual tasks, while most SOTA models score only 50% or lower.
• Using chain-of-thought (CoT) prompting not only failed to improve performance — it actually decreased it — though the drop was uneven across tasks.

From this, the team introduced the key concept of verbalizability: when image content can be accurately described in words, models perform well; otherwise, performance degrades sharply. For instance, OCR-style tasks perform reasonably well because they are inherently text-dense. But spatial tasks that cannot be fully verbalized — such as "starting from the second row, third column, move three steps to the right and land where?" — completely stump the models.

The Second Gap: Cognition and Hallucination — Models That "See But Don't Understand"

If perception is about "not seeing," visual cognition is about "seeing but failing to understand." The human visual system involves coordinated activity between the ventral stream (responsible for object recognition) and the dorsal stream (responsible for spatial relationships and action guidance). Tasks like mental rotation and paper folding require the brain to perform mental spatial transformations on three-dimensional objects — an ability humans gradually develop through embodied interaction from infancy. On cognitive tasks requiring perception + reasoning + image reconstruction, humans achieve roughly 80%, while SOTA multimodal models score only 30% to 50%, sometimes approaching random chance. The team found that models sometimes direct attention to the correct region of an image yet still fail to complete visual cognition tasks.

Most alarming was a series of "no-image tests":

• No-image test: When the original image was replaced with a completely irrelevant random image, model performance remained significantly above random chance — indicating that models rely primarily on text priors rather than actually observing the image.
• No-image++ test: Given a solid gray blank image, models would say "I can't see anything" and then immediately hallucinate an entire scene and produce an answer.

This echoes ongoing discussions in the text LLM field: CoT traces do not necessarily provide faithful reasoning paths — they more often serve as a "scaffold" for reasoning.

Faithful GRPO: Making Reasoning Traces More Faithful

To address this problem, the team proposed Faithful GRPO. GRPO (Group Relative Policy Optimization) is a reinforcement learning optimization algorithm introduced by the DeepSeek team — a variant of PPO that reduces training complexity by replacing value function estimation with group-relative reward computation. In multimodal reasoning settings, "reward hacking" is particularly pronounced: models learn to insert seemingly plausible visual descriptions into their reasoning text that are unrelated to the actual image, essentially "gaming" the reward function. When trained with standard GRPO using performance as the reward, accuracy improves somewhat, but reasoning traces become increasingly inconsistent — lacking visual grounding and exhibiting inconsistencies between the trace and the final answer. Faithful GRPO additionally introduces consistency rewards and grounding rewards (requiring that visual claims in the reasoning match the actual image content), and uses learned Lagrangian multiplier coefficients to balance constraints — thereby maintaining faithfulness while improving performance.

The research team's core conclusion is clear-eyed and pragmatic: RL fine-tuning offers limited help, and models are highly susceptible to reward hacking; neither model scale nor prompt engineering can significantly patch these deficiencies — these errors persist across different parameter counts, model families, and both open-source and closed-source models.

From Behavior Modeling to Content Generation: An Applied Perspective

Professor Rajiv Ratanshah from IIIT Delhi brought a different dimension to the discussion: behavior modeling. He drew on Shannon and Weaver's communication theory, which divides communication into three levels. Originally proposed by Claude Shannon and Warren Weaver in 1949, the framework decomposes information transmission into: Level A (technical — can information be accurately transmitted?), Level B (semantic — is information correctly understood?), and Level C (effectiveness — does information produce the intended behavior?). Originally developed for engineering communication systems, this framework has been widely adopted in communication studies, human-computer interaction, and marketing. He observed that as multimodal LLMs advance, machines have gotten reasonably good at the semantic level — but remain severely lacking at the effectiveness level: predicting the actual behavioral impact of content on users.

One counterintuitive finding: better-looking images do not necessarily mean better engagement. Polished images posted by official designers at companies like Adobe and Walmart might receive far fewer likes than seemingly ordinary images. The team also found that existing aesthetic evaluation metrics (such as LAION score and PickScore) correlate with human preferences at near-random levels.

To address this, the team released the EngageNet dataset containing 28 million tweets and proposed a "Large Content Behavior Model" (ICLR 2024) — mapping content and behavior into a shared language space. In follow-up work, they further drew on the "Wisdom of Crowds" concept, constructing an "Agent Factory" comprising 10 agents with different personas, whose collective judgments across diverse perspectives predict ad click-through rates — significantly outperforming a single LLM.

Helping Small Models Catch Up: Parity Aligner

Professor Anand Mishra from IIT Jodhpur focused on a practical challenge: how to help small VLMs (<4B parameters) close the performance gap with larger models (>4B parameters) without labeled data.

His team proposed the Model Parity Aligner (MPA) method. This can be viewed as an innovative variant of knowledge distillation — originally proposed by Hinton et al. in 2015, where the core idea is to have a small model (student) learn from the soft label output distribution of a large model (teacher). MPA's key innovation lies in "selective distillation": rather than learning from the full teacher output, it precisely identifies "disagreement samples" between teacher and student and performs targeted fine-tuning — similar in spirit to uncertainty sampling in active learning. The core insight: small and large models perform similarly on many tasks, with gaps appearing only on specific subsets. So it suffices to identify samples where "the large model is correct but the small model is wrong" and perform SFT on this small dataset.

MPA proceeds in three steps: a large VLM acts as a pseudo-annotator generating QA pairs; a "parity identification" phase pinpoints capability gaps; and finally the small model is fine-tuned in a targeted manner. Across multiple benchmarks including TextVQA and OKVQA, MPA consistently improved small model performance — with Qwen2-VL 2B gaining approximately 5%. Since it doesn't depend on logits, the method can also leverage closed-source models like GPT-4 as teachers.

However, as an audience question pointed out: large models may encode biases and stereotypes, and training small models only on "disagreement samples" may also inherit these biases — a limitation the method has yet to resolve.

Video Understanding and Attention Interpretability

Professor Makarand Tapaswi from IIIT Hyderabad presented work on compositionality, grounding, and interpretability. He found that even for 10-second short videos, models frequently make binding errors on "who is doing what" — for instance, confusing "a man in gray pants clapping" with "a woman in a purple shirt clapping" — because models process video "frame by frame, patch by patch" rather than understanding the scene as a unified whole.

On interpretability, the team adopted a causal intervention research approach, using the simplest "average pooling" operation on attention as an intervention — rather than relying on correlation analysis to infer whether attention is "faithful." This sidesteps the long-debated question of "whether attention weights truly reflect what the model is attending to," and instead directly measures the causal effect of interventions on downstream tasks: the model's attention to the image peaks when generating "strawberry," and peaks on text when generating a math answer. More critically, by manually boosting attention to images or text at the right moment, accuracy on visual-spatial reasoning tasks improved from 21% to 50% — demonstrating that both "blocking" and "enhancing" attention can effectively intervene in model behavior.

Medical Imaging: From Benchmarks to Clinical Deployment

Mercy's team at MSR India presented real-world deployment of radiology multimodal AI. Rather than focusing on benchmark performance, their model confronts genuine clinical challenges: confidence calibration, rare disease identification, and measurement-based diagnostics.

In a study on aortic dilation screening conducted in collaboration with Narayana Health hospital: the cardiothoracic ratio (CTR) is the ratio of the cardiac transverse diameter to the inner chest diameter on a chest X-ray (normal value ≤0.5), a classic radiological indicator for assessing cardiac enlargement; aortic dilation refers to an aortic diameter exceeding normal range (with 40mm commonly used as the threshold for the ascending aorta), and mild dilation (38–43mm) is extremely difficult to detect on chest films. Routine radiology reports flag only 12% of mild dilation cases, since such abnormalities are often not confirmed until CT imaging. Using measurement-based diagnostics via tool calls (such as computing CTR), AI demonstrates value exceeding that of radiologists in screening for these easily missed rare conditions. This is precisely the core advantage of medical AI: human radiologists, facing large volumes of routine cases, experience reduced sensitivity to low-probability anomalies due to cognitive load (the "attention tunneling" effect), while AI can maintain consistent alertness across every image and perform programmatic calculations for specific measurement indicators. The team emphasized that the real clinical value lies not in "catching common diseases" but in "not missing the rare ones that get overlooked."

Conclusion: Two Main Threads for Advancing Multimodal Intelligence

Drawing together all the presentations, progress in multimodal intelligence is advancing along two main threads:

• Input side: Redefining multimodal representations — exploring variable-rate visual tokenization, richer embeddings that better capture semantics and structure, and even building natively multimodal architectures.
• Reasoning side: Introducing grounded reasoning and verifiers, so that inference no longer relies on "text-based imagination" built on a foundation of perceptual failure.

As researchers repeatedly emphasized: the problem with current multimodal LLMs is not simply "insufficient data" — it is a deep architectural flaw rooted in being "text-centric." Bridging the perceptual and cognitive gaps between humans and models remains a long road, and one well worth sustained investment.

Key Takeaways