VisibilitySplat: Ray-Traced Occlusion Modeling and Relighting for 3D Gaussian Splatting

3D Gaussian Splatting (3DGS) shines in real-time novel view synthesis, but bringing this representation into inverse rendering—that is, disentangling materials and lighting from multi-view photos and supporting relighting—always runs into the occlusion problem. VisibilitySplat proposes a pragmatic approach: keep the efficient training and visualization pipeline of rasterized 3DGS, and attach an additional sparse visibility "oracle" to handle occlusion queries.

Why Occlusion Is the Bottleneck of 3DGS Inverse Rendering

Inverse rendering requires simultaneously solving for three coupled variables: geometry, materials (albedo, roughness, metallic), and lighting. In NeRF-style methods, volume rendering naturally integrates density accumulated along a ray, so occlusion comes "for free"; 3DGS, however, uses rasterized splatting, where mutual occlusion between different Gaussian primitives is not explicitly modeled. The fact that light comes from one direction and is blocked by another Gaussian simply doesn't exist in the standard forward pipeline.

The direct consequence of missing explicit occlusion is that the ambiguity between materials and lighting gets amplified: a dark surface could be dark because its albedo is low, or because it's occluded by other geometry. Networks tend to conflate the two during fitting, leading to noticeable deviations during relighting. This pain point is precisely the motivation behind the paper.

Inverse rendering is a classic problem at the intersection of computer graphics and vision: given several photos with known camera poses, infer the scene's geometry, surface material parameters (albedo, roughness, metallic, etc.), and lighting conditions, thereby enabling re-rendering under new lighting (relighting). It is the inverse process of forward rendering—forward rendering generates images from geometry + materials + lighting, while inverse rendering infers these three from images. The difficulty lies in the fact that these three groups of variables are highly coupled: the same pixel could be produced by a bright light source illuminating a dark material, or by a dim light source illuminating a bright material. Such ambiguities must be gradually resolved through multi-view consistency and explicit physical constraints, which is why the accuracy of occlusion modeling directly determines the quality of disentanglement.

Design of the Sparse Visibility Oracle

VisibilitySplat's core component is a point-based ray tracer, accelerated by a BVH (Bounding Volume Hierarchy), used to estimate the transmittance-weighted occlusion of Gaussian primitives under sampled lighting directions. It is trained separately as an "oracle" and queried sparsely within the inverse rendering objective.

There are two engineering trade-offs worth noting here. The first is sparsity: rather than performing full tracing for every ray and every direction, it samples only a subset when needed, avoiding directly stacking ray tracing overhead onto the entire training loop. The second is decoupling from the original pipeline: the main training loop still retains the gating mechanism of vanilla 3DGS, thus preserving advantages in interactive visualization, existing toolchains, and training efficiency. This "plug-in" design lowers the cost of adoption.

The shading part adopts a physically-based rendering (PBR) framework, using a principled BRDF based on the microfacet model, paired with low-resolution HDR environment maps as the light source representation.

BVH (Bounding Volume Hierarchy) is the standard spatial data structure for accelerating intersection tests in ray tracing. It recursively partitions the scene's geometry top-down into nested bounding boxes, forming a tree; during ray intersection, it first tests against the root node's bounding box—if there's no intersection, the entire subtree is skipped, reducing naive O(n) traversal to approximately O(log n). VisibilitySplat organizes Gaussian primitives into a BVH, enabling it to quickly identify the set of Gaussians a ray might hit in a given sampled lighting direction, and then estimate transmittance accordingly. Compared to GPU-native hardware ray tracing, point-based BVH solutions integrate more easily with 3DGS's Python/CUDA training stack, at the cost of construction and traversal overhead per query—which is the direct reason for choosing sparse invocation.

Microfacet BRDF is the cornerstone assumption of PBR: treating rough surfaces as composed of numerous ideal specular microfacets, with macroscopic reflection manifesting as the statistical result of these microfacets' orientation distribution (the Cook-Torrance model consisting of the normal distribution function D, geometric shadowing term G, and Fresnel term F). Albedo corresponds to the base color of the microfacets, roughness determines the sharpness of D, and metallic controls whether reflection leans toward colored specular or diffuse. HDR environment maps encode incident light from all directions in a high-dynamic-range panoramic image, and with low-resolution prefiltering, can approximately query irradiance from any direction, avoiding explicit modeling of scene light sources. The combination of the two makes shading results physically interpretable and allows disentangled materials to be genuinely transferred to new lighting for re-rendering.

Experimental Data and Ablation Conclusions

The paper compares against multiple categories of baselines on the TensoIR synthetic dataset, covering NeRF-based, tensor-representation, and 3DGS-based methods. Results show that VisibilitySplat achieves competitive performance on all three tasks—novel view synthesis (NVS), albedo reconstruction, and relighting—outperforming several early baselines on multiple metrics.

Ablation experiments yield two key findings. First, explicit visibility modeling makes a substantive contribution to relighting and material-lighting disentanglement; removing it causes noticeable degradation on these two tasks. Second, pure PBR supervision actually tends to favor "fitting appearance well"—performing decently on NVS—but this appearance fitting does not equate to genuine disentanglement. This finding suggests that metrics for inverse rendering tasks cannot rely solely on NVS numbers; disentanglement quality requires separate evaluation.

TensoIR is one of the commonly used synthetic benchmarks in the inverse rendering field, rendered with Blender. Scene geometry and materials are artificially constructed, thus providing precise ground truth for albedo, normals, and lighting, enabling quantitative measurement of disentanglement accuracy. It also provides tensor-decomposition (TensoRF)-style representations and NeRF-based methods as references. Note that synthetic data lacks complexities such as real camera noise, motion blur, and non-Lambertian surfaces, so a model's advantages on it may not directly extrapolate to real-world scenes—which is precisely why the paper itself lists generalization capability as a to-be-verified item.

Significance for the 3DGS Ecosystem

The most practical value of this work is finding a path that doesn't require starting from scratch. The renderers, editors, and training scripts accumulated by the 3DGS community all have their engineering value; completely switching to volume rendering or a ray tracing pipeline for inverse rendering would mean building anew. VisibilitySplat's approach treats occlusion as a pluggable query module, invoked on demand and used sparsely.

Of course, sparse querying also means occlusion accuracy is lossy, especially in scenes with complex geometry and dense mutual occlusion, where there's room for trade-off between sampling density and final disentanglement quality. Additionally, evaluation is currently concentrated on synthetic datasets, and generalization capability in real-world scenes still requires further validation.

Overall, the signal this research sends is: point-based visibility queries can effectively complement rasterized splatting. The two are not mutually exclusive—rather, they can fill in the physical correctness needed for inverse rendering while preserving 3DGS's practical advantages.

One-line Summary

VisibilitySplat uses a sparse visibility oracle to shore up the occlusion modeling gap in 3DGS inverse rendering

Paragraph Summary

VisibilitySplat addresses the material-lighting ambiguity caused by missing occlusion in 3DGS inverse rendering by attaching a sparse visibility "oracle" based on BVH ray tracing outside the standard rasterized 3DGS pipeline, estimating transmittance-weighted occlusion of Gaussian primitives under sampled lighting directions. This module is invoked sparsely on demand without altering the original training and visualization pipeline, thus preserving 3DGS's interactive efficiency and toolchain ecosystem. Shading adopts a PBR microfacet BRDF paired with low-resolution HDR environment maps. On the TensoIR dataset, the method is competitive on novel view synthesis, albedo reconstruction, and relighting tasks. Ablations show that explicit visibility modeling is key to relighting and disentanglement, while pure PBR supervision can fit appearance but may not achieve genuine disentanglement.