HORIA: New SOTA for 4D Human-Object Interaction Reconstruction from Monocular Video

Joint 4D reconstruction of humans and objects from monocular video is a key capability for understanding how people interact with their environment. However, existing methods still face two major bottlenecks when capturing human-object interactions: videos often contain composite semantic context spanning multiple interaction stages, making it difficult to track how interactions change over time; and under human-object occlusion, the interaction visual information contained in single-view observations is incomplete, resulting in geometrically inconsistent reconstructions.

To address these two problems, the research team proposes the HORIA framework. Through two core designs—motion-aware captioning and interaction-aware re-rendering—it achieves temporally coherent and geometrically accurate 4D reconstruction across a variety of interaction scenarios, and attains state-of-the-art performance.

Two Major Challenges: Semantic Composition and Single-View Occlusion

To understand the design motivation behind HORIA, we first need to see the nature of the problem clearly. A human-object interaction video is typically not a repetition of a single action, but a sequence of multiple stages such as grasping, moving, and placing. Each stage has a different semantic meaning, and the patterns of change in human pose and object pose also differ. If a unified representation is used to fit the entire video, the model easily becomes confused between different stages and struggles to characterize how the interaction evolves over time.

The other difficulty comes from occlusion. When a hand grasps a cup or the body blocks a chair, a monocular camera can only see part of the interaction. Occluded regions lack visual evidence, so the reconstruction network can only rely on priors to "guess," resulting in geometric inconsistencies—for example, the object's shape jumps before and after occlusion, or the contact surface between the human and the object cannot be aligned. These two challenges compound each other, making 4D human-object reconstruction from monocular video a highly under-constrained problem.

Monocular 4D reconstruction refers to simultaneously recovering the shapes of the human and the object in 3D space and their motion over time (the fourth dimension being time) from video captured with just an ordinary camera. Unlike multi-view or depth-camera solutions, monocular input lacks direct depth and disparity information and is inherently an ill-posed problem: the same set of 2D pixels can be explained by infinitely many 3D configurations. When the human and object come into contact and occlude each other, the visible pixels are further reduced and the constraints become weaker, forcing the network to rely on learned priors for inference—this is precisely the meaning of "highly under-constrained" in the text. Understanding this point is essential to seeing why additional novel-view supervision and semantic priors are particularly critical for the monocular setting.

Core Design 1: Motion-Aware Staged Semantic Captioning

The first key step of HORIA is to decompose the entire input video into interaction stages that are clearly localized in time and semantically distinct from one another. The research team adopts motion-aware captioning to generate a dedicated textual description for each stage.

The significance of this step lies in transforming the vague "entire video" into a structured "sequence of stages." Each stage has a corresponding textual description of what interaction is taking place between the human and the object during that stage. With such fine-grained semantic anchors, the subsequent reconstruction can establish clear temporal boundaries between stages and avoid cross-stage semantic confusion. From a technical standpoint, this is an approach that uses language as an intermediate representation to constrain temporal modeling, delegating changes that are hard to distinguish visually to the semantic level for disentanglement.

Core Design 2: Interaction-Aware Re-rendering to Compensate for the Single View

Staged descriptions alone are not enough; the geometric gaps caused by occlusion require additional supervisory signals to fill. HORIA's second design is interaction-aware re-rendering, whose pipeline can be broken down into three steps.

First, the initially reconstructed 4D human and object are rendered from novel viewpoints to obtain several novel-view images. These viewpoints do not exist in the original video, so they naturally carry information beyond the single view. Next, using a text-to-video diffusion prior conditioned on the stage textual descriptions, these rendered results are refined. The diffusion model plays the role of a "knowledge completer": based on the interaction semantics described in the text, it corrects unreasonable or missing parts of the novel-view renderings. Finally, the refined novel-view renderings serve as auxiliary supervision that feeds back into the 4D reconstruction process, compensating for the insufficiency of single-view cues.

The cleverness of this design lies in forming a closed loop: reconstruction produces novel-view renderings, the diffusion prior refines the renderings, and the refined results in turn supervise the reconstruction. The textual descriptions serve as a bridge, injecting the staged semantic information into the geometric optimization.

Text-to-video diffusion priors are a class of generative models pretrained on massive video-text pairs. They learn to generate video clips that are semantically consistent and temporally coherent given a textual description, with representative works such as Sora and Stable Video Diffusion. In HORIA, they are not used to directly generate the final result, but rather act as "judge and repairer": conditioned on the stage text, they denoise and refine the novel-view images rendered from the preliminary reconstruction, so that occluded regions take on a plausible appearance consistent with the interaction semantics. Compared with general image diffusion models, video diffusion priors additionally constrain inter-frame consistency, thereby preventing the refined novel views from jittering over time. This also explains why the staged textual descriptions must be accurate—the text is the only condition through which the diffusion model understands "what interaction should be seen."

Performance and Significance

According to the paper, HORIA can output temporally coherent and geometrically accurate 4D reconstructions across a variety of interaction scenarios, achieving state-of-the-art performance. This means the framework has a certain generalization ability across different types of interaction actions, rather than being tuned for a single scenario.

From a broader perspective, the value of HORIA goes beyond reconstruction accuracy itself. It demonstrates two reusable ideas: first, using staged textual descriptions to disentangle the composite semantics in long videos; second, using generative priors to provide supervision for novel views to alleviate the monocular under-constraint. Both points are instructive for downstream tasks that require understanding human-object interaction, such as motion capture, AR/VR content creation, and robot imitation learning. Of course, introducing the diffusion prior also brings additional computational overhead, and the quality of the textual descriptions affects the final reconstruction quality—these are factors that need to be weighed when deploying this line of work.

Novel view rendering as a supervisory signal (novel view supervision) is a common practice in neural rendering: if the 3D reconstruction is correct, images rendered from any novel viewpoint should be consistent with the real world; conversely, flaws in novel-view images expose geometric errors. Since monocular video has no real novel-view images, HORIA uses the renderings refined by the diffusion prior as "pseudo ground truth," back-propagating the error to the geometry and motion parameters of the human and object through differentiable rendering. This closed loop resembles pseudo-label distillation in self-supervision, and its effectiveness depends on the quality of the pseudo-labels; therefore, the fidelity of the diffusion prior and the accuracy of the text conditions together determine the upper bound of the final reconstruction.

One-line Summary

HORIA achieves 4D human-object interaction reconstruction from monocular video using staged textual descriptions and diffusion-prior novel-view re-rendering.

Paragraph Summary

To address the two major bottlenecks in 4D human-object interaction reconstruction from monocular video—composite semantics across multiple interaction stages being hard to track, and occlusion causing incomplete single-view information—the research team proposes the HORIA framework. The framework first uses motion-aware captioning to decompose the entire video into semantically clear, temporally well-localized interaction stages, generating a dedicated textual description for each stage to disentangle temporal semantics; then, through interaction-aware re-rendering, it renders the preliminary 4D reconstruction from novel viewpoints and, conditioned on the stage text, refines the renderings with a text-to-video diffusion prior, after which the refined novel views serve as auxiliary supervision that feeds back into the reconstruction, forming a closed loop. Experiments show that HORIA achieves temporally coherent and geometrically accurate 4D reconstruction across a variety of interaction scenarios, reaching state-of-the-art performance, and its ideas of staged semantic disentanglement and generative-prior supervision are instructive for downstream tasks such as motion capture, AR/VR, and robot learning.