FactorSplat: Controllable-Appearance Gaussian Proxy for Medical Volume Rendering with Disentangled Transfer Functions

The Long-Standing Problem in Medical Volume Rendering: Transfer Functions Baked Into the Model

In 3D visualization of medical images such as CT and MR scans, the Transfer Function (TF) plays a critical role — it determines how tissue at different intensity values is mapped to colors and opacity levels. By adjusting the TF, clinicians can highlight or hide structures like bone, soft tissue, and vasculature within the same scan.

The problem is that recent Gaussian proxy rendering methods trained on images — while fast and high-quality — share a fundamental flaw: they "bake" a fixed transfer function into the model's appearance representation during training. In other words, once training is complete, the model can only reproduce that one color scheme and visibility configuration. Switching to a different TF preset to view another tissue structure typically requires retraining from scratch. This is a clear bottleneck for clinical and research workflows that demand real-time interaction and frequent view switching.

The Core Idea Behind FactorSplat: Factorizing Appearance

FactorSplat, proposed in a recent arXiv paper, is designed specifically to address this pain point. It is a per-scene N-Dimensional Gaussian Splatting (N-DGS) proxy with a key capability: at inference time, it can accept region-specific intensity-to-RGBA curves and change appearance without retraining.

The design philosophy is right there in the name — factor (as in factorization). FactorSplat decomposes the rendered appearance into several composable components:

• Local lookup: Directly applies the color and opacity changes authored by the user (clinician or visualization expert), responding immediately to TF editing intent.
• Shared functional encoder paired with low-rank per-Gaussian factors: Learns the residual appearance response to TF variation — the subtle appearance differences that a simple local lookup cannot capture.
• Geometry and view-dependent appearance shared across presets: The model's geometric structure and view-dependent appearance are shared across different TF presets, avoiding redundant storage per preset.

This decomposition decouples the "invariant part (geometry)" from the "variable part (color and visibility)," enabling a single checkpoint to support diverse TF edits.

Visibility Control and TF-Aware Pruning

Medical visualization is not just about color — it's fundamentally about what you see and what you hide. FactorSplat preserves visibility control and TF-aware pruning, ensuring that when users switch transfer functions, they can still freely hide or reveal specific anatomical structures without losing information that might be needed later. This is especially important in diagnostic settings — a structure hidden under one preset may need to be clearly visualized under another.

Experimental Results: Outperforming Baselines Across Four Task Types

The paper evaluates FactorSplat on 7 CT and MR scan datasets, benchmarked against the region-aware VEG method. Testing covers four settings designed to probe generalization under different conditions:

• Validation: Standard test conditions
• Interpolation: Interpolating between known TFs
• Unseen composition: TF combinations not seen during training
• Out-of-distribution (OOD) edits: Edits beyond the training distribution

FactorSplat outperforms VEG on both average PSNR and changed-region error across all four settings. Across the seven scans, the average PSNR improvement over VEG ranges from 1.10 dB to 1.52 dB across these four categories. Critically, each scan requires only a single checkpoint to support all of these unseen edits without retraining — which is the core practical value of this approach.

---

PSNR (Peak Signal-to-Noise Ratio) is the most widely used objective metric in image quality assessment. Measured in decibels (dB), it quantifies the pixel-level difference between a reconstructed image and a reference image — higher is better. Because PSNR uses a logarithmic scale, a 1 dB improvement is perceptually significant, typically corresponding to a visible improvement in detail reproduction. In medical visualization, higher PSNR means sharper boundary contours and more accurate tissue gray-level gradations, which matter greatly for diagnostic tasks requiring fine structural analysis. FactorSplat's consistent improvement of over 1 dB across all four test settings demonstrates that its advantages are not limited to standard test conditions — they hold robustly even in the more challenging "unseen composition" and "OOD edits" scenarios encountered in real clinical workflows.

---

Performance: Real-Time Interaction Is Feasible

Speed is another major highlight. At $1600^2$ resolution, the cached fast renderer achieves an average of 524 FPS, with transfer function switching taking only 1.17 milliseconds. This means users can switch color and visibility presets with virtually no perceptible delay while viewing 3D medical volume data in real time — a fluid interactive experience. For radiologists or visualization researchers who need to adjust on the fly, this response speed effectively eliminates the waiting cost of TF switching.

Significance and Outlook

The value of FactorSplat lies in bridging efficient neural rendering techniques like Gaussian Splatting with the strict requirement for controllable, interactive appearance in medical visualization. Previous Gaussian proxy methods excelled in speed and image quality but sacrificed TF flexibility. By factorizing appearance, FactorSplat restores the freedom to edit transfer functions while maintaining high frame rates, and generalizes to unseen edit combinations.

Of course, as a per-scene training method, it still requires training a separate model for each scan — it is not a "train once, use everywhere" universal solution. But for the case-by-case analysis workflows common in clinical and research settings, a single training run that supports diverse interactive edits represents a genuine and practical advance. Interested readers can visit the project page at https://gaozhongpai.github.io/FactorSplat/ for more details.

Background

Gaussian Splatting is a scene representation and real-time rendering technique that has gained significant traction in recent years. Its core idea is to represent a scene's geometry and appearance using a large collection of 3D Gaussian ellipsoids, which are projected onto the image plane via differentiable rasterization and blended to produce final pixel colors. Compared to Neural Radiance Fields (NeRF), Gaussian Splatting is typically orders of magnitude faster and more readily achieves real-time frame rates — which is why it has been increasingly adopted for medical volume data visualization. However, traditional Gaussian Splatting directly binds color and opacity parameters to each Gaussian primitive. Once training is complete, these appearance parameters are fixed and cannot respond dynamically to transfer function adjustments at inference time — this is the technical root cause of the "baked-in" problem.

Low-Rank Decomposition is a widely used technique for parameter compression and generalization. The core idea is to express a high-dimensional matrix as the product of two or more low-rank matrices, dramatically reducing parameter count while preserving the principal directions of variation. The "low-rank per-Gaussian factors" design in FactorSplat draws on this principle: rather than storing a full appearance variation vector independently for each Gaussian, it shares a compact factor basis and has each Gaussian hold a small set of coefficients for combination. This captures the differentiated appearance response of individual Gaussians across different transfer functions, while preventing parameter count from scaling linearly with the number of TF presets — a key design choice that balances expressiveness with storage efficiency.