WorldRecon: Generating Large-Scale 3D Scene Meshes from Pose-Free Images

Reconstructing complete, high-quality large-scale 3D scene meshes from a set of RGB images with uncalibrated camera poses has long been one of the most challenging problems at the intersection of computer vision and graphics. It requires algorithms to faithfully reproduce the geometric details of observable regions while also plausibly completing surfaces in areas that were never captured or only sparsely observed. WorldRecon is a generative multi-view reconstruction framework proposed to tackle exactly this challenge, attempting to strike a balance between "geometric fidelity" and "generative completion."

Background: Why Large-Scale Scene Reconstruction Is So Difficult

Existing 3D reconstruction methods fall broadly into two categories. One is traditional multi-view stereo (MVS) and neural implicit surface methods, which achieve high accuracy when observations are sufficient but tend to leave large holes or degrade into blurry, fragmented surfaces when faced with occlusion, sparse viewpoints, and expanding scene scale. The other is native 3D generative models, which excel at synthesizing plausible shapes from few or even a single image but lack constraints from external geometric priors, making it easy to generate structures that conflict with real observations.

Large-scale scenes happen to expose the weaknesses of both categories simultaneously: large spatial extent, uneven viewpoint coverage, and pose estimates that themselves carry noise. WorldRecon's approach is not to pit the two against each other, but to use a coarse yet globally meaningful geometric "scaffold" to guide the generative model, keeping the generation process anchored to real observations throughout.

Multi-view stereo (MVS) is a classic 3D reconstruction paradigm: given multiple images with known camera poses, the algorithm matches pixels, estimates depth maps, and then fuses the depth maps into a point cloud or mesh. Its accuracy depends heavily on texture richness and viewpoint coverage, and it tends to fail in textureless, reflective, or occluded regions.

Neural implicit surface methods (such as NeuS and VolSDF) use multi-layer perceptrons to map spatial coordinates to signed distances or occupancy probabilities, then optimize through volume rendering. They can represent continuous, fine surfaces, but training typically requires dense viewpoints, and a single optimization usually covers only one scene, making it difficult to extend directly to large-scale environments.

Native 3D generative models (such as TRELLIS and the LRM series) instead regress 3D representations directly from images, possessing strong shape priors that can "guess" reasonable structures when observations are scarce. However, their generated results do not strictly align with input geometry and are prone to drift, scale distortion, or objects appearing out of thin air. WorldRecon seeks to make generative priors accept constraints from reconstruction-based geometry.

Core Method: Decomposing Large Scenes with a Global Scaffold

Using a Noisy Point Cloud as a Global Anchor

WorldRecon's first step is to use a reconstruction model to estimate camera poses from input images and obtain a coarse, noisy global point cloud. This point cloud does not aim for precision; its value lies in providing a spatial framework covering the entire scene. The research team partitions this global scaffold into multiple mutually overlapping spatial blocks, thereby transforming a "large-scene generation" problem into a series of manageable "local block generation" problems.

The key to this blocking strategy is the overlap design: adjacent blocks retain common regions, leaving interfaces for subsequent cross-block consistency propagation.

Adapting TRELLIS.2 to Incorporate Reconstruction-Based Conditional Control

To enable the generative model to truly leverage these imperfect geometric priors, WorldRecon designs a dedicated reconstruction-based conditional architecture on top of the native 3D generative model TRELLIS.2. When generating each spatial block, it simultaneously receives two types of input: the local point cloud crop corresponding to that block, and the image observations relevant to that block.

This conditioning mechanism allows the model to simultaneously output sparse structure, fine geometry, and PBR materials, rather than just a single surface shape. The introduction of PBR materials means the reconstruction results can directly serve rendering and simulation, rather than remaining at the geometric level alone.

TRELLIS.2 is a native 3D generative model proposed by Microsoft. Its core idea is to represent 3D assets as sparse structures (sparse voxel structures), implicit geometry fields (such as SLAT, Structured Latent), and material properties, and it supports generation conditioned on images or text. Compared to methods that output only a single mesh, TRELLIS-family models naturally provide both geometry and appearance simultaneously, which is why WorldRecon chose it as its base.

However, TRELLIS.2's native conditions typically include only images or text, lacking explicit spatial geometric constraints. WorldRecon's "reconstruction-based conditional architecture" in effect encodes point cloud crops together with corresponding image observations and injects them as additional conditions into the denoising process, so that generated results are locally "pulled back" near real observations by the point cloud. This conditioning approach sits between pure generation and pure reconstruction and can be regarded as a form of geometrically regularized generation.

PBR materials (Physically Based Rendering) refer to physically based rendering attributes, typically including albedo, metallicness, roughness, etc., enabling reconstructed assets to respond correctly to lighting in standard rendering engines and be used directly in simulation or game pipelines.

Neighbor-Aware Latent Inpainting Maintains Scene Coherence

A natural risk of block-wise generation is the appearance of seams or style discontinuities between blocks. To address this, WorldRecon introduces neighbor-aware latent inpainting, combined with a causal world generation pipeline, allowing structure, geometry, and material information to pass sequentially between overlapping blocks. The generation result of a previous block serves as context influencing subsequent blocks, thereby stringing local results into a unified whole.

This mechanism is key to extending "multi-view generative reconstruction" from individual objects or small rooms to truly large-scale scenes.

Latent inpainting draws inspiration from image inpainting: in the generative model's latent space, the latent representation of an already-generated neighboring block is treated as known context, and only the regions in the current block that overlap with or are adjacent to neighboring blocks undergo "completion-style" denoising, rather than regenerating the entire block from noise. This can significantly reduce seams and style discontinuities.

A causal world generation pipeline refers to generating blocks sequentially in spatial order (e.g., near to far, center outward), with the output of preceding blocks serving as conditions for subsequent blocks. This autoregressive unrolling is similar to causal attention in video generation; the benefit is that global consistency is naturally guaranteed by the generation order, at the cost of being difficult to parallelize and the risk of error accumulation along the chain. WorldRecon uses overlapping regions as buffers for information transfer precisely to mitigate error accumulation.

Experimental Results: Quantitative Metrics and Generalization Ability

The paper reports extensive experiments on synthetic and real indoor scenes. Compared to the strongest baseline, WorldRecon reduces Chamfer distance by 14.9% and 43.1% on the two test sets, respectively, while improving F-score by 17.3% and 12.0%.

These numbers deserve closer examination. Chamfer distance measures the bidirectional closeness between the reconstructed surface and ground truth; a decrease means more accurate geometry. An F-score improvement indicates that, at a given threshold, a higher proportion of surfaces are correctly reconstructed. Simultaneous improvement in both metrics shows that the method does not trade accuracy for completeness, but maintains reliable surface positions while filling holes. The 43.1% Chamfer distance reduction on real scenes is particularly notable, suggesting strong generalization ability from synthetic to real.

The research team attributes this advantage to "faithful and complete generative reconstruction" and emphasizes its stable performance in sim-to-real scenarios.

Chamfer distance measures the average bidirectional nearest-point distance between two point sets or surfaces: for each point on the reconstructed surface, find the nearest point on the ground truth surface, and vice versa, then average. It penalizes both "missing surfaces" and "extra structures"; smaller values are better, but its sensitivity to local details is limited.

F-score, on the other hand, counts the proportion of surface points in the reconstruction that are both accurate and recalled at a given distance threshold, and can be understood as the harmonic mean of precision and recall. A decrease in Chamfer distance indicates better overall fit; an increase in F-score indicates more surfaces correctly reconstructed within the tolerance range.

sim-to-real refers to a model's performance when transferred to real data after training or validation on synthetic data. Synthetic data has cleaner geometry and annotations, while real data carries scanning noise, exposure differences, and pose errors; therefore, a substantial Chamfer distance reduction on real scenes better demonstrates the method's robustness.

Significance and Limitations

WorldRecon's value lies in offering a scalable technical route: constraining a generative model with coarse geometric priors, breaking through VRAM and scene-scale limitations with blocking and overlap mechanisms, and then ensuring global consistency through cross-block context propagation. For applications requiring large-scale digital twins, indoor scene simulation, or immersive content production, this framework provides a complete pipeline from casually captured images to usable mesh assets.

One additional note: the method still relies on an initial pose estimate and global point cloud as a scaffold, and the quality of this upstream reconstruction directly affects the upper bound of the final result. In addition, block overlap and cross-block propagation also introduce extra computational overhead. How to maintain efficiency and consistency at larger scales and under sparser observations is a direction for future research.

The project homepage has been made public, containing more reconstruction results and implementation details: https://kszpxxzmc.github.io/WorldRecon-website/