[KongchangAI]
· 2 min read· 1,208 words

GRACE: Retrieval-Augmented Refinement for More Generalizable Medical Image Segmentation

GRACE: Retrieval-Augmented Refinement for More Generalizable Medical Image Segmentation

GRACE replaces per-organ medical segmentation refinement models with a retrieval-augmented universal refiner.

Manual correction of medical segmentation masks has been constrained by the "one model per anatomy" paradigm, with model counts growing linearly. GRACE proposes a retrieval-augmented universal refiner: a lightweight shape-aware projection head is trained on a frozen multimodal encoder, letting a shared knowledge base return validated cases as shape priors to edit input masks without retraining. Evaluated on 18 public datasets across CT/MRI/ultrasound with 7 upstream segmenters, it yields up to +0.35 DSC on weak segmenters (input DSC 0.386) and limited gains on strong ones (input DSC 0.932). Cross-modal transfer remains partly anatomy-dependent—an open problem.

Medical image segmentation models often produce masks that require manual correction. Traditional approaches require training a separate refiner for each anatomical structure, which is costly. GRACE proposes a retrieval-augmented universal refiner that edits input masks based on prior cases from a shared knowledge base, eliminating the need for structure-specific retraining.

GRACE Overview

Limitations of Traditional Refinement Methods

Automated segmentation is the first step in clinical quantitative analysis pipelines, but predicted masks often contain structural errors that require manual review and correction. Existing automated refinement solutions are typically designed for a single anatomical structure (e.g., liver, kidney), and each new segmentation target demands training a new refiner, causing the number of models to grow linearly with the number of targets. This "anatomy-specific" paradigm is not only expensive to maintain but also poorly suited to multi-organ, multi-modal clinical scenarios.

GRACE's core idea is to use a single universal refiner trained across multiple anatomical structures, leveraging a retrieval mechanism to obtain shape priors from similar cases in a knowledge base, thereby dynamically adapting to different segmentation tasks.

DSC (Dice Similarity Coefficient) is the most commonly used overlap metric in medical image segmentation. It is computed as twice the intersection of the predicted mask and ground-truth mask divided by the sum of their pixel counts, ranging from 0 to 1, with values closer to 1 indicating higher overlap. It essentially measures the degree of match between two sets, is more sensitive to large-volume structures, and responds relatively slowly to localized errors in small lesions. In refinement tasks, researchers often use the change in DSC before and after refinement (ΔDSC) to quantify the gain from a refiner. Therefore, the input DSC directly determines the upper and lower bounds of the refinement space—the worse the input, the greater the potential improvement.

How GRACE Works: Retrieval-Augmented Mask Editing

GRACE is a retrieval-augmented refiner that edits input masks under the context provided by a shared knowledge base. The knowledge base consists of validated historical cases, and the refiner is trained once on a multi-organ corpus containing compact organs, cavities, branching vessels, and tumor lesions.

Specifically, GRACE adds a lightweight shape-aware projection head on top of a frozen multimodal encoder, mapping embedding similarity to mask shape consistency. As a result, the knowledge base can return cases that are shape-similar to the current input, and the refiner uses these cases as shape priors to guide mask editing. This design allows the refiner to avoid retraining for new anatomical structures—simply adding validated cases to the knowledge base is sufficient for expansion.

Retrieval augmentation originates from the RAG paradigm in natural language processing: rather than hard-coding all knowledge into parameters, the model retrieves relevant entries from an external knowledge base at inference time as context, thereby gaining new information without updating weights. In GRACE, the knowledge base stores manually validated historical cases and their masks, and retrieval is based on shape embedding similarity rather than text semantics. This combination of "frozen model + scalable index" yields a key property: when adding new organs or new datasets, one only needs to append cases to the knowledge base without retraining the model itself, avoiding catastrophic forgetting and allowing the knowledge base to continuously accumulate clinical experience.

The projection head and shape consistency are key to understanding why retrieval works. Embeddings from deep encoders are typically optimized for segmentation tasks and may not preserve shape information suitable for similarity comparison. GRACE attaches a lightweight projection head on top of the frozen encoder that regresses mask shape consistency from embedding similarity, effectively "calibrating" the embedding space into a shape metric space. This way, nearest-neighbor retrieval returns cases that are genuinely shape-relevant rather than merely similar in appearance or texture. Because the encoder is frozen and only the projection head is trained, the parameter count and training cost are both low, and multimodal inputs (CT, MRI, ultrasound) can share the same retrieval structure.

Experimental Evaluation: Generalization Across Datasets and Anatomical Structures

The paper evaluates GRACE on 18 public datasets covering three modalities (CT, MRI, and ultrasound), using 9 refinement configurations and 7 upstream segmenters (spanning weak, medium, and strong performance ranges). Each refinement result is measured by the DSC change relative to the input mask.

In the held-out dataset transfer evaluation, three held-out datasets query a fixed knowledge base built solely from training data. A separate leave-one-anatomy-out experiment tests parameter-free index expansion: validated cases of an unseen solid organ are added to the knowledge base while the refiner remains frozen.

Results show that on weak segmenters, with an input DSC of 0.386, GRACE improves DSC by up to +0.35; on strong segmenters, with an input DSC of 0.932, the average change is small. This indicates that GRACE is most effective on low-to-medium quality masks, with limited refinement on high-quality inputs.

The leave-one-anatomy-out experiment draws on the idea of leave-one-out cross-validation in machine learning, but the object under test is not a sample but an anatomical category: a certain organ is deliberately excluded during training and knowledge base construction, and at test time its validated cases are added to the knowledge base while the refiner remains frozen, examining whether the system can generalize to unseen structures through retrieval alone. This setup directly mirrors the clinical reality of "new tasks constantly emerging" and distinguishes two types of generalization capability—parameter-level generalization (the model has seen similar structures) versus retrieval-level generalization (transferring solely via similar priors), the latter being the core advantage GRACE advocates.

Challenges in Multimodal and Cross-Anatomy Scenarios

GRACE's transfer performance on imaging modalities absent from both training and indexing is partial and anatomy-dependent, remaining an open research direction. For example, the model's performance on ultrasound may vary by organ, indicating that the generalization capability of retrieval-augmented refinement still has room for improvement.

Nevertheless, the results demonstrate that retrieval-conditioned refinement is a practical alternative to anatomy-specific refinement models. The code is open-sourced on GitHub for community reproduction and further research.

Practical Implications and Future Directions

GRACE's universal design reduces the need to train separate refiners for each anatomical structure, promising to lower the engineering burden of clinical deployment. Its parameter-free index expansion property allows incorporating cases of new organs without retraining, offering a new approach to continual learning.

Future work could explore how to improve the robustness of cross-modal transfer and how to extend retrieval-augmented refinement to more clinical scenarios. For medical image analysis researchers, GRACE demonstrates the potential of retrieval mechanisms in segmentation post-processing and warrants further attention.

One-line Summary

GRACE replaces per-organ medical segmentation refinement models with a retrieval-augmented universal refiner.

Paragraph Summary

Manual correction of medical image segmentation masks has long been constrained by the "one model per anatomical structure" paradigm, where the number of models grows linearly with the number of targets. GRACE proposes a retrieval-augmented universal refiner: a lightweight shape-aware projection head is trained on a frozen multimodal encoder, enabling a shared knowledge base to return validated cases as priors based on shape similarity, thereby editing input masks without retraining. The study evaluates on 18 public datasets, three modalities (CT/MRI/ultrasound), and 7 upstream segmenters, achieving up to +0.35 DSC improvement on weak segmenters (input DSC 0.386) and limited gains on strong segmenters (input DSC 0.932). Cross-modal transfer remains partially anatomy-dependent and is an open problem.

#GRACE#medical image segmentation#retrieval-augmented refinement#mask refinement#Dice Similarity Coefficient#multimodal segmentation
Share:

Related articles