DaFISH: Spherical Harmonic Rotation Decouples Lighting to Improve Novel View Synthesis Quality

In scenarios where active light sources are rigidly bound to the camera (such as camera flashes, underwater robot lighting, and endoscopic surgery), the light direction changes with the viewpoint, causing neural rendering methods (NeRF, Gaussian Splatting) to incorrectly encode this view-dependent lighting as surface color, resulting in appearance inconsistency and hindering the recovery of scene geometry and albedo. To address this, researchers propose DaFISH, a lighting disentanglement method that can be plugged into any novel view synthesis (NVS) pipeline. By representing lighting with spherical harmonics (SH), it decomposes each point's emitted color into surface albedo and SH lighting factors, achieving stable joint optimization and supporting analytical generalization to arbitrary camera poses. On held-out frames across five scenes, DaFISH achieves the highest average PSNR (38.91 dB), SSIM (0.9613), and LPIPS (0.0582), outperforming the strongest splatting baseline and the closest NeRF baseline, while maintaining scene consistency across different viewpoints.

Rendering Challenges Posed by Active Light Sources

In many real-world capture scenarios, the light source is rigidly attached to the camera and moves with it. Under this configuration, the illumination direction for each surface point changes across viewpoints. Traditional neural rendering methods (such as NeRF and Gaussian Splatting) encode this pose-dependent lighting as surface color, leading to color inconsistency across viewpoints in rendered results and masking the true surface albedo and geometry. For example, in dark-environment imaging, a flash moves with the camera, so the same surface is illuminated from different directions at different viewpoints. If the model simply memorizes colors, it cannot distinguish intrinsic albedo from lighting effects, thereby affecting the quality of novel view synthesis.

DaFISH is designed precisely to solve this ill-posed problem. By disentangling lighting from albedo, it enables the model to separate view-independent surface properties from multi-view images, thereby improving rendering consistency and geometric reconstruction accuracy.

Novel View Synthesis (NVS) aims to generate images of a scene from arbitrary unseen viewpoints given a set of input images with known camera poses. NeRF and 3D Gaussian Splatting are the two mainstream NVS methods: the former implicitly represents the radiance field with a multi-layer perceptron, while the latter explicitly represents the scene with Gaussian ellipsoids. However, these methods typically assume that lighting is view-independent (i.e., the same surface point has constant color across viewpoints). When the light source is rigidly bound to the camera and lighting changes with viewpoint, the model incorrectly bakes lighting effects into colors, causing appearance inconsistency in novel views. DaFISH corrects this assumption by explicitly modeling the dependence on light direction, and can thus be seen as a necessary extension of existing NVS pipelines for active lighting scenarios.

Spherical Harmonic Lighting Decomposition and Differentiable Rendering

The core of DaFISH is to decompose each point's emitted color into surface albedo and a lighting factor represented by spherical harmonics. Spherical harmonics are a set of orthogonal basis functions defined on the sphere that can compactly represent directional lighting. During training, DaFISH directly evaluates SH lighting, enabling stable joint optimization of albedo, lighting, and geometry. Since the conversion from SH to irradiance has a closed-form solution, the entire training pipeline from SH evaluation to volume rendering is fully differentiable, facilitating end-to-end optimization.

A key advantage of this approach is analytical generalization: after training, DaFISH can analytically compute lighting for arbitrary camera poses without retraining. This means the model can generalize to unseen viewpoints while maintaining lighting consistency, which is particularly important for practical applications such as endoscopic navigation.

Spherical Harmonics (SH) are a set of orthogonal basis functions defined on the sphere, analogous to a generalization of Fourier series on a circle. In computer graphics, SH are commonly used to efficiently represent low-frequency directional lighting: an environment lighting function can be projected onto SH bases and compactly approximated with only a few coefficients (e.g., 9 coefficients for order 2). DaFISH leverages this property to parameterize the incident lighting received at each surface point with SH coefficients, thereby modeling changes in light direction as rotations and combinations of SH coefficients rather than directly memorizing colors. Due to the orthogonality of SH bases, the product of lighting and albedo can be analytically computed in the rendering equation, providing the mathematical foundation for joint optimization.

Differentiable rendering means that every stage of the rendering pipeline (geometric transformation, lighting computation, volume integration, etc.) is differentiable with respect to scene parameters, enabling end-to-end optimization via gradient descent. The key to DaFISH is that the conversion from SH to irradiance has a closed-form solution and the integral in volume rendering is differentiable, so albedo, SH lighting coefficients, and geometric parameters can be updated simultaneously through backpropagation without staged training or additional supervision.

Experimental Results and Performance

On held-out frames across five scenes, DaFISH achieves the highest average PSNR (38.91 dB), SSIM (0.9613), and LPIPS (0.0582), outperforming all compared methods, including the strongest splatting baseline and the closest NeRF baseline. These metrics indicate that DaFISH has a clear advantage in rendering quality, especially in handling pose-dependent lighting, where it can more accurately recover scene appearance.

Furthermore, DaFISH maintains scene consistency across different viewpoints, reducing appearance inconsistency compared to prior methods. Experiments also demonstrate that DaFISH can seamlessly integrate with existing NVS pipelines (such as NeRF), validating its generality and effectiveness.

Application Prospects and Limitations

DaFISH is applicable to any scenario requiring handling of active light sources rigidly bound to the camera, such as consumer photography, underwater imaging, and medical endoscopy. Its pluggable nature makes it easy to integrate into existing neural rendering frameworks to improve rendering quality. However, the method currently targets static scenes, and extension to dynamic scenes or complex lighting changes (such as multiple moving light sources) requires further research. Additionally, the choice of SH order may affect the ability to capture lighting details and needs to be balanced in practice.

Overall, DaFISH provides an effective solution for novel view synthesis under active lighting, advancing the application of neural rendering in real-world scenarios by disentangling lighting through spherical harmonic rotation.

One-line Summary

DaFISH uses spherical harmonics to disentangle lighting from albedo, solving rendering inconsistency caused by active light sources moving with the camera.

Paragraph Summary

DaFISH is a lighting disentanglement method that can be plugged into any novel view synthesis pipeline, addressing the issue of light direction varying with viewpoint in scenes where active light sources are rigidly bound to the camera. It represents lighting with spherical harmonics and decomposes each point's emitted color into surface albedo and SH lighting factors, achieving stable joint optimization of albedo, lighting, and geometry. Since the conversion from SH to irradiance has a closed-form solution, the entire pipeline is fully differentiable and supports analytical generalization to arbitrary camera poses without retraining. On held-out frames across five scenes, DaFISH achieves an average PSNR of 38.91 dB, SSIM of 0.9613, and LPIPS of 0.0582, outperforming the strongest splatting and NeRF baselines while maintaining cross-view consistency. The method is suitable for flash photography, underwater imaging, and endoscopy, but currently targets static scenes.