CCNU's CoReFuse-Med: Purify Before Fusing to Solve Multimodal Medical Segmentation Failures

Multimodal fusion is often assumed to mean "the more information, the more accurate," but in medical imaging this assumption frequently fails. Central China Normal University, together with Huazhong University of Science and Technology, Guangzhou University of Chinese Medicine, and the University of North Carolina at Chapel Hill, proposes CoReFuse-Med. Its paper, When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation, has been accepted to ACM MM 2026. The core conclusion is straightforward: when modalities are spatially aligned but their sampling quality differs greatly, fusion provides no gain—instead, the high-quality modality gets "contaminated" by the resampling artifacts of the low-quality modalities.

A Counterintuitive Starting Point: Metrics Drop After Fusion

The paper runs a simple comparison on the IH EPVS dataset. A high-resolution single-modality T1 achieves a DSC of 0.703, but after fusing T1, T2, and FLAIR, the DSC drops to about 0.695.

These numbers highlight an easily overlooked premise: spatial alignment does not equal consistent information quality. After resampling, low-resolution sequences have the same dimensions as T1, with pixel-to-pixel correspondence—but the reliability carried by each pixel is not equivalent. The network sees "tidy" input but receives a "mixed" signal.

The authors define the scope narrowly—discussing only quality degradation caused by resolution differences, not modality misalignment or complete modality missingness. This restraint actually makes the analysis more focused.

DSC and HD95: Two Core Metrics for Medical Segmentation

DSC (Dice Similarity Coefficient) measures the overlap between predicted segmentation and ground-truth annotation, ranging from 0 to 1, with values closer to 1 indicating better overlap. It is the most commonly used segmentation accuracy metric. HD95 (95th Percentile Hausdorff Distance) measures segmentation quality from the perspective of boundary distance: it computes the 95th percentile of distances between predicted and true boundaries, typically in millimeters or voxels, with smaller values indicating tighter boundary fit. The two are complementary—DSC is sensitive to interior filling, while HD95 is sensitive to boundary shifts—so the paper reports both. A drop from 0.703 to 0.695 may seem small, but in medical image segmentation, declines of this magnitude often indicate systematic deviations at lesion margins or small structures, with clinical significance.

Three Chains of Evidence: Contamination Starts in Shallow Layers, Bias Solidifies in Deep Layers

The study does not stop at the phenomenon of "metrics dropped" but traces the failure mechanism through three paths: shallow features, gradients and occlusion, and skip connections.

Evidence 1: Low-Resolution Resampling Injects Noise in Shallow Layers

The authors propose the Feature Contrast-to-Noise Ratio (F-CNR), which divides the difference between feature means in lesion and background regions by the background standard deviation to measure feature-level separability. Visualization of third-layer features shows F-CNR of 1.0283 for T1, 0.4301 for T2, and only 0.2980 for FLAIR—T1 is about 3.4× that of FLAIR. Geometrically the three coincide, yet the reliability of shallow representations differs several-fold.

Evidence 2: Severe Asymmetry Between Gradient Contribution and Inference Impact

In the EPVS 3D U-Net, T1 accounts for 77.4% of gradient contribution, while FLAIR accounts for only 13.1%. Yet occluding FLAIR causes a 70.2% DSC drop, comparable to occluding T1. Low gradient updates do not mean no influence at inference. The authors view this contrast as core evidence of optimization–inference inconsistency.

Why Gradient Contribution and Occlusion Experiments Reveal Modality Importance

Gradient contribution is typically measured by the proportion of gradient magnitude produced by each modality's input branch during backpropagation, reflecting how much the network relies on each modality during training. Occlusion experiments, in contrast, zero out or replace a modality's input with noise at inference and observe the performance drop, reflecting that modality's influence at inference time. The two characterize modality importance from different angles: the former is a byproduct of optimization, the latter a direct causal intervention in forward inference. The paper finds that FLAIR's gradient contribution is only 13.1%, yet occluding it drops DSC by 70.2%, indicating the network rarely updates the FLAIR branch during training but relies heavily on its complementary information at inference. This asymmetry means gradient statistics cannot serve as a proxy for modality importance, or the actual contribution of low-gradient modalities would be underestimated.

Evidence 3: Skip Connections "Infect" Downstream Layers

Skip connections in U-shaped networks are intended to pass spatial detail to the decoder, but in anisotropic MRI they also carry resampled structural artifacts all the way down. The network also tends to rely on the "cleaner," easier-to-optimize high-quality modality, so feature contamination and greedy modality selection occur simultaneously.

Method Design: Decontaminate Shallow Layers, Correct Bias in Deep Layers

CoReFuse-Med does not simply stack stronger cross-attention; instead, it splits fusion into two stages while retaining a three-stream, symmetric U-shaped segmentation architecture.

LACE + IMSH: Blocking Contamination at Shallow Layers

LACE uses local DoubleConv plus strip convolution branches along the D, H, and W directions to preserve detail and capture axial context respectively, then performs selective fusion via adaptive weights generated by Softmax.

IMSH uses parameter-free 3×3×3 average pooling to decompose features into a structural base and a residual: the base is assigned competitive Softmax weights to select reliable anatomical structures, while the residual is filtered through a shared Sigmoid gate to remove resampling artifacts, finally integrated back with lightweight DSConv3D. The smooth component carries the shared anatomical layout, while the residual contains both useful detail and noise—this decomposition is key to "purification."

Motivation for Strip Convolution and Structure–Residual Decomposition

Strip convolution expands elongated kernels along a single axis (D, H, or W), capturing long-range context in that direction with few parameters; running three directions in parallel approximately covers the global structure of anisotropic volumetric data while preserving local detail. IMSH's average-pooling decomposition rests on a classic assumption: the low-frequency smooth component corresponds to shared anatomical layout, while the high-frequency residual contains detail and noise. Since resampling artifacts mainly manifest as high-frequency structural distortion, decomposing features and applying competitive selection to the base and gated filtering to the residual can suppress artifacts without additional parameters. DSConv3D (depthwise separable convolution) integrates the result back, further controlling computation, consistent with the lightweight design of only 2.665M parameters on WMH.

MBCF: Recalibrating Modality Contributions in Deep Layers

MBCF operates on deep skip-connection features at L3–L4. It first concatenates global descriptors from the three modalities and passes them through a shared MLP and Sigmoid to produce conditioned channel weights—note that Softmax is deliberately not used to force the three modality weights to sum to 1, avoiding zero-sum competition. A spatial gate then excludes unreliable responses with unstable locations, yielding a common query Q that belongs to no single modality, which performs symmetric cross-attention over K and V obtained from concatenated and downsampled tri-modal features.

The goal is not to give all modalities equal say, but to assign trustworthy information a reasonable contribution during fusion.

Why Sigmoid Instead of Softmax for Modality Weights

Softmax forces all modality weights to sum to 1; when one modality's weight rises, the others must fall, creating zero-sum competition—unreasonable when modality quality differs greatly, since a high-quality modality should not have its weight compressed by the presence of low-quality ones. Sigmoid activates each modality independently, allowing all modalities to simultaneously receive higher or lower weights, thus avoiding the trade-off. Moreover, MBCF's common query Q belongs to no single modality but is a conditioned product of tri-modal global descriptors, so cross-attention retrieves each modality's K and V based on the "fused overall need" rather than letting one modality dominate the query—helping correct the network's greedy dependence on high-quality modalities in deep layers.

Experimental Results and Boundaries to Note

On EPVS, Ours(Med) achieves DSC 0.7326 and HD95 7.35, outperforming MedNeXt's 0.7085 and 8.17; on BraTS, DSC 0.8528 and HD95 4.43, also better than MedNeXt's 0.8314 and 7.59; on WMH, DSC 0.80, HD95 6.22, IAVD 0.171, with only 2.665M parameters.

Ablation studies show that on EPVS the Baseline achieves DSC 0.7146, while combining LACE+IMSH+MBCF brings Ours(Base) to 0.7322, indicating the three modules complement each other.

Two boundaries deserve attention. First, the two BraTS experiments retaining 20% and 30% axial slices report the best DSC/HD95 obtained by retraining each model separately, not by testing the same weights across degradation levels; the additional Gaussian noise experiments serve only as supplementary validation. Second, in severely degraded regions of EPVS, the model's role is mainly to prevent contamination from destroying T1, not to "magically recover" information already lost in T2/FLAIR.

The research boundaries are also clear: it focuses on resolution degradation after spatial alignment, not complete modality missingness or all severe-noise scenarios; future directions include dynamic modality missingness and more acquisition artifacts.

A Takeaway for Multimodal Fusion Practice

The most valuable aspect of this work is not that it refreshes a few segmentation metrics, but that it breaks down "why fusion fails" into observable, intervenable steps. For medical imaging engineers, there are at least two direct implications: first, modality quality assessment should be a prerequisite step in the fusion pipeline, rather than assuming alignment equals usability; second, training-time statistics such as gradient contribution cannot serve as proxy indicators of inference-time importance.

Both the paper and code are publicly available, providing a reproducible baseline for handling imbalanced modality quality in multimodal medical segmentation.

One-line Summary

CoReFuse-Med reveals the fusion failure mechanism in multimodal medical segmentation and proposes a two-stage decontamination and bias-correction scheme.

Paragraph Summary

Central China Normal University and collaborators propose CoReFuse-Med, challenging the default assumption that "multimodal fusion necessarily helps." The paper shows that in medical images that are spatially aligned but differ greatly in sampling quality, fusion actually lets high-quality T1 be contaminated by resampling artifacts from low-quality modalities—on EPVS, single-modality DSC of 0.703 drops to about 0.695 after fusion. Through three chains of evidence—F-CNR feature separability, asymmetric gradient contribution vs. occlusion, and artifact propagation via skip connections—the study locates the failure mechanism and designs LACE and IMSH to decontaminate shallow layers and MBCF to recalibrate modality contributions in deep layers via symmetric cross-attention with a shared query. On EPVS, BraTS, and WMH, both DSC and HD95 outperform MedNeXt, with only 2.665M parameters on WMH. The paper also clarifies boundaries: BraTS degradation experiments are results from separately retrained models, and the model's role is to prevent contamination rather than recover lost information.