Multimodal and Embodied Intelligence: A Comprehensive Look at Progress, Pitfalls, and Opportunities

At the "Multimodal and Embodied Intelligence" session of the Microsoft Research (MSR) India academic summit, experts including Professor Madhav Krishna from IIIT Hyderabad, Professor Sriram Ganapathy from the Indian Institute of Science (IISc), and Dr. Somak Aditya from IIT Kharagpur shared their research practices across robotics, healthcare, speech, and multimodal reasoning, followed by an in-depth panel discussion. This cross-modal, cross-domain exchange revealed the true state of multimodal AI today—its progress, its overlooked pitfalls, and its future opportunities.

Robotics: How Classical Methods and End-to-End Models Play to Their Strengths

Professor Madhav Krishna used a vivid metaphor—"how we cook our own kichadi (an Indian rice-and-lentil medley)"—to describe his lab's core methodology: rather than blindly embracing end-to-end large models, they combine foundation models with the classical robotics stack, drawing on the strengths of each.

He began by outlining the value of the classical navigation stack: building maps via SLAM, relocalizing through localization frameworks, computing trajectories with global planners, and continuously localizing using Kalman filters or Monte Carlo methods. The greatest advantage of these modules lies in their interpretability—they operate directly in Cartesian and task space, representing scenes with human-understandable point clouds, making them relatively easy to debug.

To understand why these "classical" techniques remain revered by robotics researchers today, it helps to understand the core problems each of them solves. SLAM (Simultaneous Localization and Mapping) addresses the robot's chicken-and-egg dilemma: to localize itself it needs a map, and to build a map it needs to know its own position—SLAM solves both simultaneously through online joint optimization, making it a foundational technology for autonomous mobile robots, with variants relied upon by everything from robotic vacuum cleaners to self-driving vehicles. The Kalman filter is a recursive Bayesian estimation algorithm that minimizes state estimation error by fusing sensor measurements (such as IMU and GPS) with motion-model predictions; because it provides an optimal closed-form solution under linear-Gaussian assumptions, it has been widely adopted in aerospace and robotics control for decades. When systems exhibit nonlinear characteristics (such as a wheeled robot turning), Monte Carlo Localization is used instead—based on particle filtering, it represents the probability distribution of the robot's pose using a large number of randomly sampled "particles," better handling multi-hypothesis scenarios and capturing multimodal distributions. MPC (Model Predictive Control) is an online receding-horizon optimization control strategy: at each time step it solves an optimization problem over a finite horizon while explicitly handling hard constraints such as obstacle avoidance and joint torque limits—precisely the properties that end-to-end neural networks struggle to guarantee reliably, making MPC the controller of choice for robotics scenarios that require safety assurances.

By contrast, while end-to-end models promise "input at one end, output at the other," they face fundamental challenges. Madhav put it incisively: "A classical framework actually solves the problem every time it faces it, whereas an end-to-end model just does a single forward pass." This means the latter is highly data-dependent, constrained by its training distribution, and struggles with the hard constraints ubiquitous in robotics tasks (for example, a robotic arm must never collide with a monitor). Moreover, in low-compute, low-data scenarios, running a massive foundation model on a moving drone is extremely difficult in itself.

CrowdSurfer, SparseLoc, and Smart Wheelchairs

Based on this philosophy, the team developed a series of hybrid systems: CrowdSurfer uses vision-language models paired with optimal control; SparseLoc uses a VLM to locate objects in a scene and then completes localization with Monte Carlo methods; and LegoDrive combines small language models with differentiable optimization to achieve goal-directed navigation.

Building these systems was enabled by the rapid maturation of the multimodal foundation-model ecosystem in recent years. CLIP (Contrastive Language-Image Pre-training), introduced by OpenAI in 2021, maps images and text into a shared embedding space through contrastive learning, giving models zero-shot image classification and cross-modal retrieval capabilities and fundamentally transforming the research paradigm for joint vision-language understanding. Grounded DINO combines DINO (a self-supervised vision Transformer) with open-set object detection, enabling it to locate arbitrary objects in an image via natural-language descriptions without requiring separately labeled training data for each object class. SAM (Segment Anything Model), developed by Meta, can perform zero-shot segmentation of any object in an image; its core innovation is framing segmentation as a promptable, interactive problem. LLaMA is Meta's series of open-source large language models, ranging from 7B to 70B parameters, whose open weights have made it an important foundation for academic research and resource-constrained deployment. Together, these models form the "building-block layer" of today's multimodal AI applications, powering composite systems like SparseLoc that require no end-to-end training—developers can invoke off-the-shelf capabilities like assembling building blocks, without training a massive model from scratch, dramatically lowering the barrier to developing AI robotics applications.

A typical case is the smart wheelchair. When a user issues a complex verbal command like "I need to put away some hardware first, then store books and place clothes," the LLM parses out the key phrase "hardware," aligns it via CLIP encoding with visual encodings, identifies a table in the SLAM map as the placement location, and—using LiDAR-to-camera extrinsic calibration—maps the table into a 3D point cloud, with the classical MPC stack ultimately driving the wheelchair to navigate there. The entire pipeline trains no models at all, using only off-the-shelf foundation models.

In SparseLoc, the team uses models such as LLaMA, Grounded DINO, and SAM to identify static landmarks in a scene (like tree trunks, shrubs, and traffic lights), compressing a four-million-point point-cloud map into a sparse representation of just 4,000 centroids for localization and navigation—an innovation that classical methods alone could not achieve.

Medical AI: The "Last Mile" Challenge of Deployment

Professor Sriram Ganapathy shared the work of the IISc AI Center of Excellence in Health (AI CoE in Health), which he leads. Funded by the Ministry of Education of India's central government, it is a three-to-four-year project of 300–500 million rupees, whose core mission is to bring AI to a broader segment of society.

He cited a study published in JAMA, pointing to a harsh reality: despite the success of large language models and chatbots in many domains, major AI success stories in healthcare remain few and far between. The reason cuts to the heart of the matter—models are developed without adequately considering the deployment phase. "We develop models using academic metrics or easily codable metrics, but once you take them into the field, all sorts of other metrics come into play."

The center focuses on care delivery for non-communicable diseases (NCDs), choosing to deploy AI at the first point of contact where patients enter the healthcare system—because this is where population coverage is widest and resource constraints are most severe. Sriram repeatedly emphasized that all solutions are "AI-assisted" rather than "AI-replacing," and that they are not patient-facing; instead, they empower other roles in the healthcare system such as nurses, junior doctors, and ASHA workers.

Oral Cancer Screening: A Maturing Edge AI Application

The oral cancer workflow is one of the more mature applications: a mobile AI model captures an image of the mouth and determines whether there are suspicious lesions requiring further attention. The model was trained on 3 million images from 30,000 subjects, over more than two years of collection and validation, and is currently undergoing Class B licensing approval from India's drug regulator, the CDSCO.

Two background dimensions worth exploring lie behind this case. On the regulatory side, the CDSCO (Central Drugs Standard Control Organisation) is India's federal body responsible for regulating medical devices and software; its Class B license corresponds to medium-risk medical software and AI-assisted diagnostic tools. The approval process requires submitting clinical validation data, software lifecycle documentation (following the IEC 62304 standard), and post-market surveillance plans, and the entire cycle typically spans several years—which explains why such a long gap exists between an AI medical product's technical maturity and its actually reaching patients. On the technology selection side, MobileViT is a lightweight vision Transformer architecture proposed by Apple's research team, designed specifically for mobile and edge devices: it combines locally processing convolutional layers (which excel at capturing spatial locality and are parameter-efficient) with globally processing Transformer blocks (which excel at modeling long-range dependencies), achieving efficient inference on low-compute devices with a model footprint just a fraction of a standard ViT. This is especially critical for grassroots healthcare scenarios: ASHA (Accredited Social Health Activist) workers in rural India cover roughly 900 million rural people, typically using low-end smartphones in areas with weak network coverage. Offline operation capability directly determines whether an AI solution can truly reach those who need it most, rather than remaining confined to urban tertiary hospitals.

This MobileViT-based model runs in just 30 seconds, requires no cloud access, and can operate offline on the edge devices of health workers' phones.

In addition, Sriram introduced his lab's work on uncertainty modeling—SURE (Semantic Uncertainty and Reliability Estimation). The core idea is to enable a model not only to make a decision but also to express its confidence, candidly saying "I don't know" when evidence is insufficient, thereby proactively abstaining in high-error-rate scenarios.

To understand the significance of SURE, one must appreciate the inherent flaw of deep learning models in expressing uncertainty. Standard neural networks typically output point estimates (such as softmax probabilities), but these "probabilities" are often overconfident—a model may give a wrong answer with 95% confidence. Researchers distinguish two sources of uncertainty: epistemic uncertainty stems from insufficient training data and reflects a model "knowing what it doesn't know," which can in principle be reduced by collecting more data; aleatoric uncertainty stems from noise inherent in the data itself (such as a blurry image or an inherently ambiguous lesion boundary) and cannot be eliminated by adding more data. Mainstream estimation methods include: MC Dropout (randomly dropping neurons during inference, with the variance over multiple forward passes serving as an uncertainty estimate), Deep Ensembles (training multiple independent models and aggregating their disagreement), and Post-hoc Calibration (such as temperature scaling, which rescales logits so that output probabilities align with actual accuracy). SURE further models both types of uncertainty at the semantic level. In medical AI, the Selective Prediction mechanism—declining to give an answer under low confidence and referring the case to human review rather than forcing an output—has been shown to reduce clinical risk more effectively than blind output. When reviewing AI medical devices, regulators such as the FDA and CDSCO have already made uncertainty quantification and abstention mechanisms important review dimensions.

This is especially critical in high-stakes fields like healthcare.

Multimodal Reasoning: The Perceptual "Foundation" Is Not Yet Solid

Dr. Somak Aditya's talk focused on video question answering and multimodal reasoning, and his conclusions carry a cautionary message. His team started with educational video QA, attempting to have multimodal large language models (MLLMs) automatically answer student questions based on lecture videos.

However, the research revealed a core problem: factual recall accuracy is extremely low, and even supervised fine-tuning struggles to improve it. More surprisingly, when the team separately tested "recall ability," they found that even the highly popular GPT models performed quite poorly—across all categories, including spatial information, transcribed content, physics experiments, and fine-grained prioritization, only about 50% of facts were correctly recalled, with open-source models faring even worse.

From this, Somak drew a profound point: "If you can't even get perception right, reasoning is out of the question." He used well-structured visual planning puzzles (such as defining initial and goal states in the PDDL planning language) to isolate perception from reasoning problems, and found that a large number of errors arose precisely from incorrect perception of the initial and goal states.

PDDL (Planning Domain Definition Language) is the standard formal language in the field of AI planning, originating from the 1998 AIPS planning competition, used to precisely define state spaces, the preconditions and effects of legal actions, and the search for a sequence of actions from an initial state to a goal state. The unique value of using PDDL to build benchmarks lies in its formal precision: because state-transition rules are entirely described by symbolic logic, there is no ambiguity, allowing researchers to precisely distinguish two independent capability dimensions—"can the model correctly perceive the current visual state" and "can the model reason from a correct current state to a plan for reaching the goal." This decoupling of capabilities is crucial for diagnosing the causes of multimodal model failures: in traditional end-to-end evaluations, a wrong final answer could stem from perception errors, planning errors, or both, with no way to tell them apart. The findings of Somak's team show that many failure cases the research community attributes to "insufficient reasoning ability" actually originate from misreadings at the perceptual level—the model can't even correctly describe the color and position of blocks in an image, let alone plan. This finding has important directional implications for how to prioritize the allocation of research resources.

Enhancing Fine-Grained Perception with Diffusion Models

To address this problem, the team explored a visual representation method fusing diffusion models with the CLIP architecture. They observed that CLIP-type models excel at capturing high-level semantics but sacrifice fine-grained detail, whereas diffusion models can better capture that fine-grained information.

This insight comes from the research community's recent rediscovery of the discriminative capabilities of diffusion models. Diffusion Models were originally designed for generative tasks, learning the data distribution by progressively adding noise to data and training a network to learn the reverse denoising process; representative models include DALL-E 2, Stable Diffusion, and Imagen. However, a recent line of work (such as DIFT and Emergent Correspondence) has found that the activation features in the intermediate layers of a diffusion model's U-Net architecture naturally carry rich spatial and texture information—intuitively natural, since to accurately reconstruct every pixel of an image, the model must internally maintain fine-grained representations of color, material, and shape boundaries. Experiments show that these features even outperform discriminative models trained specifically for classification on discriminative tasks such as semantic correspondence and fine-grained object matching. This stands in stark complement to CLIP: CLIP's contrastive pretraining objective encourages aligning an entire image with an entire passage of text into a single vector (global semantic matching), making it inherently insensitive to local details; whereas the skip-connection layers at different depths of a diffusion model's U-Net separately encode hierarchical information from low-level textures to high-level semantics. By extracting patch-level features at different levels and fusing them with CLIP's CLS global features, researchers can simultaneously obtain the complementary advantages of both models: global semantic understanding + fine-grained local perception.

Through clever combinations of features at different levels, patch-level features, and CLS features, the team achieved a 2–5% improvement on visual tasks, a 4–6% improvement in vision-text alignment, and even a 4–12% improvement in open-vocabulary segmentation.

Panel Discussion: AGI, Scaling, and Lessons from Failure

In the panel session, the three experts engaged in a candid exchange on the big-picture questions of multimodal AI.

Is the bottleneck in algorithms or hardware? Madhav shared his conversation with Bill Gates—Gates believes the bottleneck is hardware, citing that the human hand has 27 degrees of freedom, while a robotic hand demonstrated by a Chinese startup has reached 22. Madhav's judgment was "both": in navigation, hardware problems are essentially solved, but in areas like dexterous manipulation and aggressive drone flight, hardware remains a long-term obstacle. He also particularly emphasized the importance of deployment capability: "The era of just being able to write code is over; today people expect you to actually deploy in a warehouse, on a search-and-rescue drone."

How should we understand AGI? All three experts held a generally cautious stance. Sriram believes the concept of AGI is riddled with ambiguity, and that AI is fundamentally still about replacing, augmenting, or simulating human intelligence. Vineet offered a brilliant "Matrix" metaphor: we live between two worlds, virtual and physical; most current definitions of AGI revolve around the online world we are immersed in, but once we step out of this "Matrix" and walk in a real room, almost no AI can help us navigate.

The essence of the deployment pitfall. Madhav's view was sharp: "The ML work is the easy part; dealing with people is what's extremely hard—people are messier than models." He cited Wadhwani's newborn anthropometry project as an example: an ASHA worker uses a phone costing ten thousand rupees to shoot a video that measures a newborn's weight. The real value lies not in the model itself, but in the data being fed back to community health officials in real time, rather than arriving six weeks later.

Scaling Is Not the Only Answer

On the question of "which problems don't depend on scale and are suitable for academia-industry collaboration," the experts unanimously favored the directions of small language models and task-specific agents. Sriram emphasized the value of goal-directed models tailored to specific use cases; Madhav pragmatically noted: "If a small model can do what a large model can do, I don't see why you'd throw three dollars per million tokens at the task you care about." He called for a return to fundamentals: "At the end of the day, we're identifying problems and solving them."

The discussion finally touched on an intriguing definitional dispute—"What actually counts as a foundation model?" When various models such as SAM3 and VGGT all call themselves "foundation models," has the term already been overused? Vineet also pointed out a fundamental limitation in the development of multimodal large models: most architectures are just "a fixed text LM with an encoder bolted on for each modality," where all other modalities accommodate the way text is processed—which is itself a deep bottleneck.

This critique touches on the core design-philosophy question of current multimodal AI architectures. Mainstream multimodal large models (such as GPT-4V, LLaVA, and Gemini) almost all follow the "LLM hub + modality-specific encoder" architectural paradigm: a visual encoder (usually CLIP-ViT) compresses an image into a fixed number of token sequences, which are then aligned to the LLM's text token space via a linear projection layer or a Q-Former (such as the query Transformer in BLIP-2), with the text LLM ultimately processing all modalities uniformly. The fundamental limitation of this design is that information from modalities like images and audio must be "translated" into a format the text LLM can digest, and this translation process itself loses information that cannot be expressed as token sequences—such as precise spatial relationships and continuous audio temporal structure. True multimodal fusion should occur at the early feature level, with cross-modal modulation (such as cross-modal attention mechanisms that let visual features directly influence the computation of language features), rather than mere token concatenation at the top layer of the LLM. The output space problem is equally fundamental: text output is a discrete sequence of symbols, whereas continuous action control, pixel-level image generation, and audio waveform synthesis require entirely different loss-function designs. Under a unified next-token-prediction framework, how to provide high-quality training signals for non-text modalities—for example, the gradient-scale discrepancy between the regression loss for continuous values and the cross-entropy loss for discrete tokens—remains an unresolved engineering and theoretical challenge. This is why researchers believe that, until we truly understand the multimodal output space, current architectures are merely "good enough to work" rather than "optimally designed."

He argued that we have not yet truly understood the output space of multimodal tasks, and this directly determines the design of loss functions and backpropagation.

Key Takeaways