Multimodal Embodied Agents: PAPAV, a Unified Capability Framework from Computer Use to Robot Use

Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, with computer-use agents being the most typical manifestation; robot systems (RSs), by contrast, couple perception and execution under physical dynamics. When multimodal agents move from virtual environments into the physical world, a core question emerges: what exactly changes when going from computer use to robot use? A recent paper proposes the PAPAV unified capability framework to answer this question.

Defining Multimodal Embodied Agents: Goal-Directed Physical Interaction Systems

The paper defines multimodal embodied agents (MMEAs) as: goal-directed systems that combine multimodal task reasoning with physical actions and revise decisions based on feedback. The key to this definition lies in "coupling"—not just the combination of perception and reasoning, but the closed loop between reasoning and physical execution.

Unlike pure software agents, MMEAs must produce actions in the real world and bear the irreversible consequences those actions entail. This raises the paper's core question: when agents migrate from computer use to robot use, which capabilities remain unchanged, and which need to be redesigned?

Partial Observability is a classic setting in reinforcement learning and robotics: the agent cannot directly know the full state of the environment and can only obtain local information through noisy sensor observations. In computer-use scenarios, agents can typically read the complete DOM tree or screen pixels, so the state is relatively transparent; in the physical world, robots are constrained by camera field of view, occlusion, sparse tactile sensing, and other factors, and must maintain uncertainty estimates (such as probability distributions or belief states) over unobserved parts. This constraint directly determines that the perception module needs active exploration and information fusion capabilities rather than passive input reception.

The PAPAV Framework: Five Recurring Core Capabilities

The paper proposes the PAPAV framework, which divides capabilities by function rather than architecture. The five capabilities are:

• Perceive: acquire multimodal information about the current state
• Anticipate: predict the possible effects of actions
• Plan: formulate a feasible course of action
• Act: complete the task through an interface or a body
• Verify: confirm whether the result meets expectations

The value of this division is that it allows MMEAs, MMAs, and RSs to be compared within the same coordinate system. For example, the "Act" of a computer-use agent is realized through API or GUI operations, whereas the "Act" of a robot involves joint motion and physical contact. The function is the same, but the implementation cost differs vastly.

Anticipate differs from planning in time scale and causal depth: planning focuses on "how to reach the goal," while anticipation focuses on "what an action will trigger." In robotic manipulation, anticipation involves reasoning about physical dynamics—for example, predicting liquid sloshing when grasping a cup or predicting obstacles behind a door when pushing it. Anticipation for computer-use agents is usually based on discrete state transitions (the page jumps after clicking a button) and can be precisely simulated with rules or world models; physical anticipation, however, must handle continuous states, contact mechanics, and material uncertainty, with errors accumulating along action chains. This is precisely why anticipation is rarely evaluated in existing benchmarks: it is difficult to measure with a single success-rate metric.

Five Physical Constraints: Fewer Presets, Fewer Fallbacks

The paper identifies five physical constraints—from partial observability to unverifiable outcomes—which together lead to two consequences: fewer pre-fixed choices and less room for error correction.

In computer-use scenarios, agents can undo operations, retry workflows, and roll back states; in the physical world, a spilled cup cannot be "Ctrl+Z"-ed. Partial observability means the agent can never obtain the complete world state and must make decisions under incomplete information. Unverifiable outcomes mean that the success or failure of some tasks cannot be judged at the moment and can only be answered by time. These constraints directly shape the open challenges proposed later in the paper.

Irreversibility is one of the most essential differences between physical interaction and software operation. Software systems naturally support undo stacks, snapshot rollback, and idempotent retries, allowing agents to trial and error at low cost; once a physical action is executed, its consequences (such as spilling liquid or collision damage) cannot be eliminated through state rollback. This requires agents to establish safety boundaries and risk assessment mechanisms before Act, and also makes "exploration" costly in physical scenarios. The paper lists unverifiable outcomes and irreversibility as constraints precisely because together they compress the room for error correction: actions cannot be undone, and results may appear with delay, so agents must make high-risk decisions under incomplete information.

Evaluation Status: Among 64 Benchmarks, Anticipation and Verification Are Almost Absent

The paper reviews 64 benchmarks and finds a striking imbalance: explicit evaluation concentrates on Act capability, with only 5 benchmarks assessing Anticipate and only 3 assessing Verify. Anticipation and verification capabilities are mostly hidden behind task success rates and are not measured separately.

This means the existing evaluation system may overestimate the true capabilities of agents. A robot successfully grasping an object does not mean it anticipated the chain reactions after grasping, nor does it mean it verified the stability of the grasp. The paper therefore calls for: evaluation should cover all five capabilities, not just task success rates.

Task Success Rate is the most commonly used aggregate metric in robot learning benchmarks, usually defined as the proportion of trials in which a preset goal is completed. It is concise and comparable, but it has attribution blind spots: the success rate does not distinguish whether the agent "succeeded by chance" or "succeeded after anticipating and verifying." For example, in grasping tasks, the success rate cannot reflect whether the agent evaluated post-grasp stability, nor can it distinguish perception errors from planning failures. The paper notes that only 5 benchmarks assess Anticipate and 3 assess Verify, precisely because these capabilities require process-level annotation or counterfactual evaluation, whereas existing benchmarks mostly use end-to-end outcome scoring, causing capability dimensions to be obscured by a single metric.

Open Challenges and the Significance of PAPAV

The five physical constraints also frame the open challenges identified in the paper: how to perform robust anticipation under partial observability? How to establish safety boundaries before irreversible actions? How to design verifiable physical interaction protocols?

The significance of the PAPAV framework lies in providing a common foundation—both for designing reliable physical agents and for measuring progress across capabilities. It reminds researchers and engineers that going from computer use to robot use is not a simple scenario transfer, but a systematic restructuring of capability weights and constraints.

The paper's core contribution is not proposing a new model, but providing a mirror that lets the field see clearly where it has overinvested and where it has underinvested.

One-line Summary

The paper proposes the PAPAV framework to compare capability differences and physical constraints between computer use and robot use.

Paragraph Summary

The paper proposes the PAPAV unified capability framework, dividing the core capabilities of multimodal embodied agents into five functions: perception, anticipation, planning, execution, and verification, and uses this to compare computer-use agents, multimodal agents, and robot systems. The migration from virtual to physical brings five physical constraints, resulting in fewer preset choices and less room for error correction, with partial observability and irreversible consequences particularly prominent. A review of 64 benchmarks shows that evaluation is heavily biased toward execution capability, with only 5 assessing anticipation and 3 assessing verification; anticipation and verification are obscured by task success rates. The paper calls for evaluation to cover all five capabilities and for PAPAV to serve as a common foundation for designing reliable physical agents and measuring progress.