HORIA

The HORIA framework, through two core designs—motion-aware captioning and interaction-aware re-rendering—achieves temporally coherent and geometrically accurate 4D reconstruction across a variety of interaction scenarios.

HORIA achieves state-of-the-art performance on the task of 4D reconstruction of human-object interaction from monocular video.

HORIA uses motion-aware captioning to generate a dedicated textual description for each interaction stage.

HORIA's interaction-aware re-rendering pipeline consists of three steps: rendering the preliminary 4D reconstruction of the human and object from novel views, using a text-to-video diffusion prior to refine the rendered results conditioned on the stage text, and feeding the refined renderings back as auxiliary supervision for the 4D reconstruction.

HORIA uses a text-to-video diffusion prior, conditioned on the stage text descriptions, to refine the novel-view renderings.

HORIA uses renderings refined by the diffusion prior as pseudo-ground truth, back-propagating the error to the geometry and motion parameters of the human and object through differentiable rendering.

HORIA's closed loop resembles pseudo-label distillation in self-supervision, and its effectiveness depends on pseudo-label quality; the fidelity of the diffusion prior and the accuracy of the text conditions together determine the upper bound of the final reconstruction.

Introducing the diffusion prior brings additional computational overhead, and the quality of the text descriptions also affects the final reconstruction quality; these are factors that must be weighed when deploying this approach in practice.

The HORIA framework has reference value for downstream tasks that require understanding human-object interaction, such as motion capture, AR/VR content creation, and robot imitation learning.