Coarse3D: Controllable and Faithful Sparse Structure Generation for Improved Image-to-3D Quality

The field of image-to-3D generation has seen a new breakthrough. The Coarse3D model improves the sparse structure generation stage to achieve direction-controllable voxel geometry that is highly consistent with the input image, laying a solid foundation for subsequent fine-grained generation.

Background: Why Sparse Structure Matters

In the coarse-to-fine 3D-native generation pipeline, the sparse structure produced in the first stage determines the global orientation and overall geometry of the object, and is critical to the quality of the final 3D model. However, existing coarse generators suffer from two major pain points: first, due to ambiguous semantic canonicalization, the generation direction is difficult to control; second, weak 2D-3D grounding leads to structures that are inconsistent with the input image. Coarse3D is proposed precisely to address these two problems.

According to the paper, Coarse3D can generate direction-controllable and structurally faithful voxel geometry from arbitrary pose-free images, and can directly replace the coarse stage in existing pipelines.

The coarse-to-fine generation paradigm is a common strategy in 3D content generation: first generate a low-resolution or low-detail global structure (the coarse stage), then progressively refine local geometry and texture (the fine stage). This divide-and-conquer approach significantly reduces the difficulty and computational cost of directly generating high-precision 3D models. In 3D-native generation, the coarse stage typically outputs sparse representations such as voxels, point clouds, or implicit fields, and its quality directly determines whether subsequent refinement can converge to a reasonable result. Therefore, the coarse stage must not only ensure geometric plausibility but also align semantically and spatially with the input image; otherwise, errors will be amplified during the refinement stage. Coarse3D's improvements focus precisely on this pivotal intermediate step.

Three Core Designs

Coarse3D introduces three key designs to improve generation quality:

1. First-frame-anchored canonical space: Aligns the generation process with the first input view while preserving object-center normalization, thereby enabling explicit control over the generation direction.

2. Dual-tower DiT: Couples generation with an auxiliary feed-forward reconstruction, learning features with spatial grounding and strengthening the association between 2D and 3D.

3. Spatial-prior-guided attention: Regularizes cross-attention using pixel-to-voxel correspondences, further ensuring consistency between the generated structure and the input image.

These designs work together to give Coarse3D significant improvements in both sparse structure quality and final 3D generation results.

DiT (Diffusion Transformer) is a generative model that introduces the Transformer architecture into the diffusion model backbone, typically taking patch sequences as input and modeling global dependencies through self-attention. Compared with U-Net-based diffusion models, DiT has advantages in scalability and long-range relationship modeling. The "dual-tower DiT" in this paper refers to running two Transformer towers in parallel: one responsible for generation (denoising), and the other serving as an auxiliary feed-forward reconstruction branch; the two share or interact features, thereby providing the generation process with spatial supervision signals from the input image and enhancing 2D-3D grounding. This coupled design is equivalent to adding a "reconstruction constraint" pathway alongside the generation backbone, which helps suppress structural drift.

Cross-attention is a mechanism in Transformers that lets one sequence (e.g., 3D voxel tokens) query another sequence (e.g., 2D image features), and is commonly used in conditional generation. In image-to-3D tasks, cross-attention is responsible for injecting image information into the 3D generation process, but its attention distribution often lacks spatial constraints, which can easily lead to structural misalignment. Spatial-prior-guided attention uses the geometric correspondences between pixels and voxels (such as camera projection) to construct prior masks or biases that regularize cross-attention, making each 3D token focus more on the region in the image that corresponds to its spatial position. This is equivalent to providing "geometric anchors" for attention, thereby improving the fidelity of the generated structure to the input image.

Experimental Performance: A Win-Win in Quality and Consistency

Experiments in the paper show that Coarse3D leads by a large margin in sparse structure generation quality and can consistently enhance the final 3D generation results, achieving state-of-the-art performance. Since Coarse3D can directly replace the coarse stage of existing pipelines, its improvements are plug-and-play, and it is expected to advance the practical application of image-to-3D generation technology.

Significance and Outlook

Coarse3D addresses the issues of direction control and structural consistency in sparse structure generation, providing a more reliable intermediate representation for image-to-3D generation. Its design approach also suggests that by introducing spatial priors and auxiliary reconstruction tasks, 2D image information can be effectively injected into the 3D generation process. In the future, this technology may be extended to more complex scenes and dynamic object generation.

(Note: This article is based on publicly available paper information; please refer to the original paper for specific details.)