SUPRA-TEX: Super-Aligned Textures for High-Fidelity Single-Image 3D Generation

Generating material-bearing, relightable 3D assets from a single image has long been one of the most challenging tasks at the intersection of computer graphics and generative AI. The core conflict lies in the need to faithfully reproduce the visual details from the reference image while plausibly completing surface materials in unobserved regions. Existing methods often fall into a dilemma—multi-view fusion tends to produce artifacts, while implicit feature conditioning struggles to preserve fine textures. The newly proposed SUPRA-TEX framework addresses this by decoupling visible material estimation, surface alignment, and observation-guided 3D completion, achieving SOTA performance while training only on public datasets.

Bottlenecks of Existing Methods and SUPRA-TEX's Breakthrough

Traditional single-image 3D texture generation typically follows two paths: one is multi-view diffusion models, which generate textures from multiple views and fuse them onto a 3D mesh; the other is conditioning methods based on implicit image features, directly injecting reference image features into the 3D generation network. The former is prone to seam artifacts when materials are inconsistent across views, while the latter loses high-frequency details due to feature compression. The authors of SUPRA-TEX point out that the root of the problem lies in modeling "what is seen" and "what is not seen" together, causing the generator's capacity to be diluted.

The core design philosophy of SUPRA-TEX is division of labor: letting the 3D generator handle only two things—faithfully preserving evidence in visible regions and plausibly completing unobserved surfaces. Visible material estimation and surface projection alignment are delegated to specialized modules. This decoupling strategy allows limited 3D training data to be focused on the most critical generation tasks.

PBR (Physically Based Rendering) materials are the standard representation in modern 3D content creation, typically composed of a set of parameterized maps, including albedo (the intrinsic color without lighting), roughness, metallic, normal, and other channels. Unlike directly storing RGB colors, PBR explicitly separates the optical properties of materials, enabling rendering engines to recompute surface light response under arbitrary lighting conditions, thus achieving relightability. This is precisely the goal of SUPRA-TEX—the generated asset is not just a texture with a fixed appearance, but a set of materials that can correctly behave under new light sources and viewpoints. Albedo and shadow separation is key to PBR prediction, as reference images often contain lighting information from the capture; without separation, this lighting gets "baked" into the material, leading to incorrect appearance after relighting.

Three-Stage Pipeline: From Image Editing to Flow Transformer

Stage 1: Lighting-Separated PBR Map Prediction

First, the research team trained an image editing model based on a multimodal large language model (MLLM) to predict lighting-separated PBR material maps at the reference view. This means the model outputs albedo, roughness, metallic, and other channels, rather than simple RGB textures. This step separates "material estimation" from 3D generation, leveraging the rich pre-training knowledge in 2D image editing.

Stage 2: Projection and Native Observation Latent Encoding

The predicted PBR maps are projected onto visible surfaces, then encoded via a pre-trained material VAE encoder into a "native observation latent encoding" that shares the same latent space as the generation target. This step is key to achieving "super-alignment"—the observation latent encoding and the generation latent encoding are in the same space, avoiding information loss from cross-space alignment.

VAE (Variational Autoencoder) is often used in generative AI as a bridge to compress high-dimensional images or material maps into low-dimensional latent spaces. The so-called material VAE is an encoder-decoder specifically trained for PBR material maps, learning a compact latent space where similar material textures are close to each other and can be manipulated by the generation network. SUPRA-TEX makes the observed PBR maps share the same latent space as the generation network, meaning "observations extracted from the reference image" and "complete materials to be generated by the network" use the same encoding convention, allowing direct concatenation or interleaved processing without additional cross-space mapping training. Cross-space alignment causes information loss because features learned by different encoders often have inconsistent semantics and scales, requiring an extra projection network that may smooth out high-frequency texture details.

Stage 3: Interleaved Flow Transformer for Complete Material Generation

The generation stage employs an interleaved flow transformer. It uses the observation latent encoding as fixed, read-only anchors, alternating between updating unobserved regions and updating the entire surface, ultimately generating the complete material latent encoding. This design ensures "what you see is what you get" fidelity in visible regions while maintaining coherent completion of unobserved surfaces.

Flow matching is an emerging generative modeling paradigm that can be seen as an important variant of diffusion models. Instead of stepwise noise addition and denoising, it learns a velocity field that continuously transforms a noise distribution into a data distribution, with a more direct training objective and more flexible sampling paths. A flow transformer combines this continuous transformation with the Transformer architecture, using attention mechanisms to model dependencies on sequences or graph structures. The "interleaved" in SUPRA-TEX refers to alternating two types of updates during generation: one modifies only unobserved regions, and the other uniformly corrects the entire surface, while fixing the observation latent encoding as read-only anchors. Anchors do not participate in updates, acting as constant evidence constraints for the generation process, keeping visible regions faithful to the reference image and preventing the network from "forgetting" or altering observed content during completion.

Performance: Trained on Public Data, Competitive with Commercial Systems

SUPRA-TEX is trained solely on public datasets but achieves SOTA on both geometry-conditioned texturing and end-to-end asset generation. The paper claims that on in-the-wild images, its results match or even surpass leading commercial systems. The key lies in the decoupled design, which allows the model to focus on alignment and completion rather than learning material estimation from scratch.

From a technical perspective, SUPRA-TEX contrasts sharply with concurrent multi-view fusion approaches. The latter rely on large amounts of 3D data to train multi-view consistency, while SUPRA-TEX achieves higher fidelity with less 3D data through native latent space alignment. Of course, this approach heavily depends on the material estimation accuracy of the 2D editing model; if the reference image has extreme lighting or occlusion, errors in the projection stage may propagate to the final result.

Implications for the 3D Generation Field

SUPRA-TEX's value is not just in刷新ing metrics but in validating a design principle: decouple generation tasks from perception tasks and use mature 2D capabilities to empower 3D generation. As MLLM image editing and material VAEs continue to advance, this "super-alignment" idea is expected to extend to more complex 3D generation tasks such as dynamic scenes and interactive assets. For developers and researchers, the project page is already public and worth following for its subsequent performance on larger-scale data.

Background Supplement

Super-alignment is a core concept proposed by SUPRA-TEX, referring to representing and directly aligning observation signals and generation targets in exactly the same latent space. Traditional approaches condition the generation network on the reference image, but reference image features and generation features often come from different encoders and different spaces, creating a semantic gap during conditioning. Super-alignment shares the material VAE encoder, directly converting projected observations into a representation isomorphic to the generation latent encoding, bypassing cross-space mapping and reducing information loss. The transferability of this idea lies in: as long as there is a sufficiently strong 2D perception or editing model and its output can be encoded into the same representation as the target generation space, mature 2D capabilities can be "transported" into 3D generation without relying on expensive 3D supervision.

One-line Summary

SUPRA-TEX decouples material estimation, alignment, and completion to achieve SOTA single-image relightable 3D asset generation using public data.

Paragraph Summary

SUPRA-TEX proposes a decoupled framework for the challenge of generating material-bearing, relightable 3D assets from a single image. Existing methods face issues: multi-view diffusion fusion tends to produce seam artifacts, and implicit feature conditioning loses high-frequency textures, because they model "visible content" and "invisible completion" together. SUPRA-TEX splits the task into three stages: first, an MLLM-based image editing model predicts lighting-separated PBR material maps; second, these are projected onto visible surfaces and encoded by a material VAE into "native observation latent encodings" in the same space as the generation target; finally, an interleaved flow transformer uses the observation latent encodings as read-only anchors, alternately updating unobserved regions and the entire surface to generate complete materials. Trained only on public datasets, it achieves SOTA on geometry-conditioned texturing and end-to-end asset generation, matching or even surpassing commercial systems on in-the-wild images, validating the design principle of empowering 3D generation with mature 2D capabilities.