Rethinking Sub-Volume Sampling Strategies in 3D Medical Self-Supervised Learning

Random Cropping: The Overlooked Bottleneck in 3D Medical Self-Supervised Learning

In self-supervised learning (SSL) for 3D medical imaging, the pretraining stage almost universally relies on random sub-volume sampling. This practice seems simple and efficient, but it implicitly carries a questionable assumption: that all spatial regions are equally important and equally worth observing.

The reality of medical imaging tells a different story. Human anatomy has inherent spatial priors, and significant spatial redundancy exists across different regions. In other words, not every cropped sub-volume carries equal value for representation learning. Random cropping completely flattens this intrinsic inequality.

This study begins by questioning exactly that ingrained practice — in 3D medical SSL, "where to look" may matter far more than we have assumed.

The core idea behind SSL in medical imaging is to leverage large amounts of unlabeled data through pretraining tasks, enabling models to learn meaningful representations that can then be transferred to downstream tasks with scarce annotations (e.g., organ segmentation, lesion detection). 3D medical volumes such as CT and MRI scans are large and typically cannot be fed into a GPU in their entirety, so pretraining commonly involves cropping sub-volumes: extracting a number of small 3D patches from the original volume as input. This practice was inherited from random crop augmentation in 2D natural image settings, where it works well. However, human anatomy in medical imaging follows highly regular spatial distributions — bones, organs, and blood vessels all occupy predictable locations — meaning sub-volumes from different positions vary significantly in their semantic value. Patches that capture air, fat, or the outer margins of bone, for instance, contribute almost nothing to learning clinically relevant representations, yet they account for a non-trivial fraction of training computation.

Better Sub-Volumes Lead to Better Downstream Transfer

The research team first conducted a set of controlled "crop quality manipulation" experiments to validate a central observation: higher-quality sub-volumes lead to better downstream task transfer performance.

This finding may seem intuitive, but it carries important methodological implications. It establishes that the sampling strategy — the observation policy — is a long-overlooked performance bottleneck in 3D medical SSL. Prior work has mostly focused on improving model architectures, loss functions, or augmentation strategies, while giving little attention to the most upstream component: where to sample from.

Given how critical the observation policy turns out to be, the authors naturally reframe sub-volume sampling as a "where-to-look" problem, rather than a simple exercise in random coordinate selection.

From Random Cropping to a Probe Bank: Method Design

Building on these insights, the study proposes a plug-and-play framework that replaces traditional random cropping with an offline coarse-to-fine probe bank.

The construction of this probe bank is guided by three core principles:

• Anatomy-awareness: Sampling should respect the distributional regularities of human anatomy, prioritizing anatomically plausible regions.
• Informativeness: Preference is given to sub-volumes that are information-rich and more valuable for representation learning.
• Diversity: Repeated sampling of similar regions is avoided, ensuring broad spatial coverage.

To examine the contribution of different levels of prior knowledge, the authors design two variants:

Lightweight Variant: Geometric and Image-Statistical Cues

The lightweight version relies solely on geometric information and image-statistical cues to construct the probe bank. The advantage of this design is its low computational overhead and the absence of any need for additional semantic models — it falls entirely within the category of volume-intrinsic priors.

Diagnostic Variant: Frozen Visual Features

A second, diagnostic variant uses frozen visual features to introduce richer semantic cues. Its role is primarily to address a research question: are lightweight intrinsic priors already sufficient, or can richer semantic information reliably provide additional value?

This contrastive design makes the framework not just an engineering solution, but also an analytical tool for investigating the fundamental nature of sampling strategies.

"Coarse-to-fine" is a common hierarchical processing strategy: a rough selection is first made at low resolution or low computational cost, and the results are then refined at higher resolution or cost. In the context of probe bank construction, the "coarse" stage uses anatomical priors and image statistics to quickly filter out low-value regions (e.g., air outside the body, homogeneous backgrounds), while the "fine" stage further selects final sub-volume coordinates from the candidate pool based on informativeness and diversity. This design allows the probe bank to be built offline once before pretraining begins; during training, sub-volumes are sampled directly from the stored coordinates with virtually no additional online computational overhead. "Plug-and-play" means the framework does not alter the SSL backbone's architecture or loss function — it only replaces the sampling module — and can therefore be seamlessly integrated into any existing SSL pipeline such as MAE, SimCLR, or DINO.

"Frozen visual features" refers to using a pretrained visual model (e.g., a ViT or ResNet trained on large-scale natural or medical images) to extract features, without updating any of its parameters during probe bank construction (hence "frozen"). The extracted features serve solely as a scoring signal to measure the semantic richness of sub-volumes, and do not participate in the backpropagation of the main SSL training. The purpose of introducing this variant is to establish a "semantic information upper bound" baseline: if even frozen features carrying rich semantic information fail to significantly outperform the lightweight variant, it suggests that geometric and image-statistical cues already capture most of the information needed for sampling decisions, and that the marginal benefit of incorporating heavyweight semantic models is limited. This contrastive design integrates engineering improvement with scientific inquiry.

Experimental Results and Mechanistic Analysis

The study conducts extensive experiments across multiple 3D medical SSL backbones and diverse downstream tasks. Results show that the lightweight variant consistently and reliably improves transfer performance over random sampling — meaning that even without heavy semantic features, a probe bank built from geometric and image-statistical cues alone yields meaningful gains.

More valuably, the study also provides mechanistic analysis explaining the sources of the performance improvement, attributing the gains to:

• Observed regions that are more anatomically plausible and information-rich;
• Significantly reduced spatial redundancy, preventing the model from repeatedly learning repetitive or low-value regions;
• More efficient learning dynamics, with smoother convergence;
• More structured and organized learned representations.

Rather than stopping at "performance is better," this analysis goes further to reveal why it is better, making the conclusions substantially more compelling.

Implications for 3D Medical AI

The significance of this work extends beyond any single specific method. It reminds the field that in the pursuit of ever-more-complex models and objective functions, the most fundamental component — how data is observed — should not be overlooked. For medical imaging, where anatomical structure is central and spatial redundancy is pervasive, where you look genuinely matters.

Because the framework is plug-and-play and the lightweight variant adds virtually no extra overhead, it has strong practical deployment potential and can be conveniently integrated into existing 3D medical SSL pipelines, offering a low-cost path to improving pretraining quality.