ROMA: A Multi-Sensory Active Perception System That Lets Large Language Models Drive Their Own Decision-Making

From Passive Perception to Active Investigation

Traditional robot perception pipelines typically follow a fixed pattern: sense, then act. The perception stack is designed as a static input channel — the robot passively receives sensor data and makes decisions based on it. Active Perception in this paradigm is usually just an add-on: the system occasionally reorients a sensor outside its normal workflow, but the core logic of "perceive first, act second" remains unchanged.

ROMA (Real-World Object-Centric Multi-Sensory Active Perception) challenges this thinking entirely. Rather than treating perception as a passive receiving stage, ROMA lets the system actively determine "what information is currently missing," then purposefully go and acquire it, while continuously tracking whether the evidence is sufficient to make a final judgment about an object's properties.

The concept of Active Perception was first systematically introduced by Bajcsy in 1988: perception should not be a passive recording of the environment, but an act in which an agent actively controls the sensor acquisition process according to its current goals. Classic implementations include active vision (directing a camera toward uncertain regions) and exploratory touch (using grasping or pressing to infer object properties). However, traditional active perception systems typically rely on predefined exploration strategies or reinforcement learning policies, making them poorly suited to open-ended tasks. ROMA's breakthrough lies in unifying all three decisions — when to explore, what to explore, and when to stop — under language model reasoning, rather than hardcoding them as fixed rules.

ROMA's Core Mechanisms

ROMA's key innovation lies in three tightly interlocked capabilities: identifying information gaps, actively acquiring information, and evaluating evidence sufficiency.

The system first analyzes which properties of a target object remain uncertain in its current understanding — such as material, weight, temperature, or surface texture that are difficult to assess from vision alone. It then actively plans how to close these gaps: adjusting its viewpoint, touching the object, or invoking other sensor modalities to collect supplementary data.

The most valuable aspect is knowing when to stop. The system tracks collected evidence in real time and only commits to an attribute estimate once that evidence is sufficient to support it. This mechanism avoids wasting resources on excessive data collection, and prevents the risk of drawing premature conclusions from insufficient evidence — bringing the perception process closer to how humans reason under uncertainty.

LLM as the Planning Core

ROMA is built around a large language model (LLM) planner. The LLM serves as the "decision-making brain" — responsible for understanding task goals, inferring the current information state, and generating the next information-acquisition action.

This design reflects an important trend in robotics and embodied intelligence: using the reasoning and planning capabilities of language models to orchestrate complex multi-step perception tasks. Compared to traditional rule-based or fixed-policy active perception approaches, the integration of an LLM brings stronger generalization and adaptability to open-ended tasks, enabling the system to "think" about what information to acquire in a way that resembles natural language reasoning.

Using LLMs for robot task planning has rapidly gained traction in recent years. Landmark works such as SayCan (Google, 2022) and Code as Policies (2023) have demonstrated that language models can decompose high-level semantic instructions into executable low-level action sequences. In active perception scenarios, LLMs are especially well-suited: perception is inherently semantic — goals like "determine whether this object conducts electricity" or "confirm whether the surface is rough" align naturally with a language model's comprehension and reasoning capabilities. At the same time, LLMs face unique challenges in embodied settings: accurately perceiving current physical states, avoiding "hallucinations" that lead to erroneous action plans, and reasoning efficiently under real-time constraints all remain open problems in this direction.

Real-World Multi-Sensory Hardware Deployment

It is worth emphasizing that ROMA does not remain confined to simulation — it is built on real multi-sensory hardware. The two key descriptors, "Object-Centric" and "Multi-Sensory," mean the system integrates data from multiple sensing modalities centered on specific objects. This is precisely the most challenging and most valuable aspect of real-world applications.

Bridging the gap from simulation to physical robots has long been one of the hardest problems in embodied intelligence. Real hardware introduces noise, latency, and unpredictable physical interactions. ROMA confronts these challenges directly, validating its active perception loop in real-world scenarios — which makes its results far more compelling.

Multi-Sensory Fusion in robotics typically involves the coordinated use of modalities such as vision, touch, force, temperature, and acoustics. "Object-Centric" means the system's perception and reasoning are organized around individual objects as the basic unit, rather than globally modeling the entire scene — this design simplifies cross-modal data alignment and makes it easier for the system to focus on task-relevant properties. Sim-to-Real transfer commonly faces issues such as perceptual noise, execution error, and contact dynamics uncertainty — collectively known in the field as the "reality gap." By building and validating the system directly on real hardware, ROMA's findings can be referenced without additional transfer engineering, and provide a more credible performance benchmark for future research.

Open-Source Code and Paper

This research releases both the complete paper and code. For researchers and developers working in robot perception, embodied intelligence, and LLM planning applications, open-sourcing means the work can be directly reproduced, validated, and extended.

At a time when "active perception" is still a frontier area of exploration, a system with real hardware implementation and publicly available code provides the community with a valuable reference baseline. It is not merely a theoretical paper — it is an engineering paradigm that practitioners can get their hands on.

Significance and Outlook

The direction ROMA represents — enabling intelligent systems to actively decide "what they need to know" rather than passively "receiving what arrives" — carries implications for future robots, autonomous driving, industrial inspection, and any scenario that requires reliable judgment in uncertain environments.

When perception is no longer a fixed preprocessing step but a dynamic, reasoning-driven investigation process, both the robustness and efficiency of systems stand to improve significantly. As LLM planning capabilities continue to advance, architectures that use reasoning to drive perception may become one of the defining paradigms of embodied intelligence.

Note: This article is based on a research summary posted on Reddit. For specific technical details and experimental results, please refer to the original paper and open-source code.