Medical Image Segmentation: Do Foundation Models Fine-tuned or Task-Specific Training Come Out on Top?

The field of medical image segmentation is undergoing a paradigm debate: on one side are foundation models pretrained on massive natural image datasets (such as DINOv2), and on the other are task-specific models trained from scratch (such as U-Net and ResU-Net). Intuitively, foundation models seem like they should win across the board, but a systematic comparative study offers a more nuanced answer.

Study Design: From Bias Tuning to Full Fine-Tuning

The study focuses on DINOv2 and designs a range of adaptation strategies, from tuning only the bias terms to full fine-tuning, comparing them against U-Net and ResU-Net trained from scratch. The experiments cover three imaging domains—colonoscopy, breast ultrasound, and dermoscopy—and the test sets include both held-out data from the training source and independent datasets from other institutions. This cross-site validation design directly addresses the weakness of most prior comparative studies, which often evaluate only within the training distribution and lack well-justified baseline configurations.

DINOv2 is a self-supervised visual foundation model released by Meta in 2023. Based on the ViT (Vision Transformer) architecture, it was pretrained on roughly 142 million curated natural images using self-distillation, without human annotations. It outputs general-purpose visual feature representations and does not include a segmentation head itself, so it must be "adapted" for downstream tasks: common approaches include bias-only tuning, training lightweight adapters (such as LoRA), or full fine-tuning. Bias tuning updates only a small number of bias parameters in the network while freezing the rest of the weights; it is a parameter-efficient fine-tuning (PEFT) strategy suited to scenarios with small datasets and limited compute. Its underlying assumption is that the pretrained features are already good enough and only a slight shift in activation distributions is needed.

Core Findings: External Generalization vs. In-Distribution Performance

The clearest conclusion is that the adapted foundation models show better generalization on every external dataset. When models need to be deployed across institutions and devices, this advantage has direct clinical value.

But within the training distribution, the story reverses. When training data is sufficient, task-specific models remain competitive and are more precise in boundary delineation, whereas foundation models more frequently miss small lesions. This shows that the foundation models' advantage does not hold unconditionally but depends heavily on whether the application scenario is "in-distribution" or "cross-distribution."

External generalization refers to a model's performance on data it has not seen during training and that comes from different institutions or devices, while in-distribution performance refers to accuracy on a held-out test set from the same source. The two often diverge: a model may memorize the textures and acquisition characteristics of the training distribution, scoring high in-distribution but dropping significantly at another site. In medical imaging, differences in scanner models, imaging protocols, and patient populations across hospitals all cause domain shift, so cross-site validation (out-of-distribution testing) reflects clinical deployability better than single-center holdout. Boundary delineation accuracy is typically measured with the Hausdorff distance or boundary F1 and is sensitive to small lesions; small-lesion misses relate to recall. Together they determine the clinical usability of a segmentation.

Adaptation Capacity: A Small Number of Parameters Suffices, and the Domain Determines the Need

The study also answers a key practical question: how many parameters of the foundation model actually need to be adapted? The answer is only a small fraction. More interestingly, the required capacity is determined by the imaging domain rather than the adaptation method. This means engineering teams deploying these models should plan their adaptation budget according to the complexity of the target domain, rather than agonizing over which fine-tuning technique to choose.

Hidden Concerns in Benchmark Datasets

The study also discloses a finding worth watching: a widely used polyp test set shares duplicate images with another benchmark. Benchmark contamination systematically overestimates model performance, especially when foundation models may have already seen related data during pretraining.

Benchmark contamination refers to overlap between samples in a test set and the pretraining corpus or another benchmark's data, so that the model has "seen the answers" during evaluation, systematically inflating performance. In the foundation model era this risk is amplified: the pretraining data of models like DINOv2 comes from large-scale web crawling, and medical images may have been inadvertently included. Detection methods include image hash deduplication, perceptual hash (pHash) nearest-neighbor retrieval, and embedding-space similarity comparison. If a polyp test set shares duplicate images with another public benchmark, then rankings and conclusions based on that test set need to be reconsidered—especially when comparing pretrained models with models trained from scratch, since the former benefits more from contamination.

Practical Implications

For teams planning to bring foundation models into medical imaging pipelines, this study provides a clear decision framework: if the goal is cross-site robustness, adapting foundation models is worth the investment; if data is plentiful and boundary accuracy is the priority, task-specific models remain a reliable choice. At the same time, teams should carefully check whether the benchmarks they use contain duplicates or contamination, and determine adaptation capacity on the target domain.

One-line Summary

A systematic comparison shows that the adapted DINOv2 generalizes better across sites, but still trails task-specific models on in-distribution small-lesion and boundary accuracy.