FastVim+Mamba Multi-Scale Fusion: A New SOTA for Prostate Cancer Gleason Grading

Gleason grading of prostate cancer is key to risk stratification, but automated grading of whole-slide images (WSIs) faces three major challenges: subtle morphological differences between grades, severe class imbalance caused by rare patterns, and diagnostic features that span multiple spatial scales. A recent paper proposes a multi-scale classifier based on state space models, achieving macro F1 0.894 and Gleason 5 F1 0.802 on the PANDA dataset—improvements of 12% and 21%, respectively, over the best single-scale model.

Problem Background: Limitations of Single-Scale Foundation Models

Pathology foundation models (FMs) can extract powerful patch-level representations, but they typically operate at only one scale. This limits spatial context, which is especially detrimental for grading categories that depend on broader context. For example, diagnosing Gleason 5 may require observing the entire glandular architecture, whereas a single-scale model can only see local texture.

More challenging still, fusing multi-scale and complementary models often requires substantial data and computational resources. The paper explicitly states that, under limited data and compute, how to effectively fuse multi-scale information remains an open problem.

Gleason grading is the core standard for pathological diagnosis of prostate cancer, proposed by Donald Gleason in the 1960s. It classifies glandular architecture into five grades, from well differentiated (Gleason 1-2) to poorly differentiated (Gleason 4-5). The final score is the sum of the scores for the most predominant growth pattern and the secondary growth pattern (e.g., 3+4=7). Gleason 5 represents the worst poorly differentiated pattern, in which glandular architecture has almost disappeared, and cells are arranged in solid sheets or individually; it occupies a very small proportion of the WSI. This rarity is precisely the root of class imbalance: models tend to overlook small but clinically significant high-grade regions. In addition, the boundaries between different Gleason patterns are blurred; in particular, distinguishing grade 3 from grade 4 depends on details such as glandular fusion and cribriform architecture, which places high demands on spatial resolution. Understanding this pathological background helps clarify why the paper emphasizes that multi-scale context is especially critical for identifying Gleason 5.

Method: Dual-Source Fusion of FastVim + Frozen Foundation Models

The authors designed a multi-scale classifier based on state space models, whose core is the fusion of two complementary feature sources:

• Frozen foundation model: provides high-quality patch representations without fine-tuning.
• Lightweight FastVim: pretrained on ImageNet and distilled from the foundation model, directly reading raw image crops.

At each scale, a Mamba model encodes features from both sources in linear time to capture spatial context; a jointly trained classification head then fuses the encoded representations into a single prediction. The key to this design is that FastVim is lightweight and fast, the foundation model provides strong semantics, and Mamba efficiently handles long-sequence spatial relationships.

According to the paper, introducing FastVim allows the model to bypass part of the computation of the heavy foundation model at inference time while retaining its knowledge.

State Space Models (SSMs) are a class of sequence modeling frameworks whose core idea is to describe an input sequence through the evolution of hidden states over time, which can be expressed mathematically as linear ordinary differential equations. Traditional SSMs have high computational complexity on long sequences, whereas Mamba, through an input-dependent selective mechanism and a hardware-aware parallel scan algorithm, reduces sequence modeling complexity to linear time while maintaining the ability to model long-range dependencies. In computational pathology, WSIs are partitioned into tens of thousands of patches, resulting in extremely long sequences; the quadratic complexity of Transformers becomes a bottleneck, and Mamba's linear complexity fits this scenario well. FastVim is a lightweight variant based on the vision Mamba architecture, designed specifically for efficient image classification. Its pretrained weights can be obtained by distillation from large foundation models, thereby greatly reducing parameter count and inference overhead while retaining semantic knowledge.

Experimental Results: Gains from Multi-Scale Fusion

Using 5-fold cross-validation on the PANDA dataset, the method achieves macro F1 0.894 and Gleason 5 F1 0.802. Compared with the best single-scale model, the relative improvements are 12% and 21%, respectively. The improvement in Gleason 5 F1 is more pronounced, indicating that multi-scale context is more helpful for identifying rare high-grade patterns.

The paper also conducted controlled experiments covering 5 foundation models, 5 base classifiers, and 3 scales. The results show that the gains hold across different encoders and are larger at smaller scales. This means the method is robust to scale variation and does not depend on a particular foundation model.

Multi-Source Cross-Validation and Implications

This study comes from an academic preprint, and no independent reproduction has yet been reported. However, its controlled experiments themselves constitute a degree of internal cross-validation: the combination of 5 foundation models × 5 classifiers × 3 scales shows that the gains are not accidental. Compared with existing single-scale methods, the improvement brought by multi-scale fusion is especially evident for categories such as Gleason 5 that depend on global structure.

One detail worth noting is that FastVim is distilled from the foundation model, which means deployment may reduce reliance on large foundation models—of practical significance for pathology AI applications with limited computational resources. The paper also acknowledges that the challenge of fusing multi-scale and complementary models under minimal data and compute has not been fully solved, but this work provides a feasible path.

Limitations and Future Directions

The paper does not report validation on external datasets, and the results on the PANDA dataset need confirmation in more independent cohorts. In addition, the distillation process may introduce biases from the foundation model, and the generalization ability of FastVim remains to be further tested. Although the computational overhead of multi-scale fusion is alleviated by Mamba's linear complexity, inference speed in real clinical workflows has not been disclosed.

Overall, this study demonstrates the potential of state space models for multi-scale fusion in computational pathology and offers new ideas for Gleason grading in resource-constrained settings.

Background Supplement

The PANDA dataset is a public benchmark released for the 2020 Kaggle competition “Prostate cANcer graDe Assessment.” It contains approximately 10,600 prostate biopsy WSIs covering six Gleason grades from 0 to 5, making it one of the largest Gleason grading datasets in computational pathology today. Its annotations were completed jointly by multiple urologic pathologists, and an ISUP grade consistency assessment was provided. Macro F1 is a metric that takes the arithmetic mean of the F1 scores across classes, assigning the same weight to minority classes (such as Gleason 5) as to majority classes, so it can sensitively reflect model performance under class imbalance; Gleason 5 F1, by contrast, separately measures recognition accuracy for the rarest and clinically most critical category. The combination of these two metrics precisely highlights the core value of the paper's method in addressing class imbalance and rare pattern recognition.

One-line Summary

A Mamba-based state space multi-scale classifier fuses frozen foundation models with lightweight FastVim, significantly improving prostate cancer Gleason grading performance.

Paragraph Summary

To address the challenges of subtle morphological differences, class imbalance, and multi-scale features in Gleason grading of whole-slide images, this paper proposes a multi-scale classifier based on state space models. At each scale, the method fuses features from a frozen pathology foundation model with features from lightweight FastVim (distilled from the foundation model); Mamba encodes spatial context in linear time, and a joint classification head outputs predictions. In 5-fold cross-validation on the PANDA dataset, it achieves macro F1 0.894 and Gleason 5 F1 0.802, relative improvements of 12% and 21% over the best single-scale model, respectively. Controlled experiments covering 5 foundation models, 5 classifiers, and 3 scales verify that the gains hold across encoders and are larger at smaller scales. The introduction of FastVim is expected to reduce deployment reliance on heavy foundation models, but the study has not yet reported external validation or clinical inference speed.