MambaXray-PRB: A New Benchmark for X-Ray Report Generation on CheXpert Plus

Radiologists face a massive volume of X-ray images every day, and writing standardized diagnostic reports is both time-consuming and mentally exhausting. Radiology Report Generation (RRG) from X-ray images is one of the core challenges that medical AI is working to solve — with the potential to significantly reduce clinicians' workload and shorten patient wait times. A recent arXiv paper proposes a systematic solution in this direction, establishing a standardized benchmark while introducing a new framework called MambaXray-PRB.

The Field's Bottlenecks: Missing Benchmarks and Poor Domain Adaptation

Despite notable progress in RRG in recent years, researchers have identified persistent bottlenecks at two levels.

The first is the lack of a standardized benchmark. The newly released CheXpert Plus dataset is of high quality, but it lacks accompanying baseline implementations and evaluation results. This means subsequent algorithms cannot be trained, quantitatively evaluated, and fairly compared under a unified standard — severely hindering cross-method comparison and iterative progress in the field.

The second is insufficient domain adaptation of general-purpose large models. While today's popular large language models and multimodal large models are highly capable, directly applying them to highly specialized fields like radiology often fails to accurately capture the correspondence between medical imaging features and report terminology. In other words, a powerful general model is not the same as a useful medical model.

Building a Comprehensive Benchmark: A Ruler for Future Work

To fill the gap left by CheXpert Plus's lack of baselines, the research team established a comprehensive benchmark covering mainstream X-ray report generation models and large language models. The value of this benchmark lies in providing a reliable basis for comparison, allowing researchers to quickly identify the state-of-the-art approaches in the field.

From a research methodology perspective, this type of benchmark work is often underestimated, but it is actually the infrastructure that drives healthy development in the field. Without a unified evaluation standard, performance numbers reported across papers are hard to accumulate into meaningful scientific progress. The team's willingness to take on this work has positive implications for the entire RRG community.

CheXpert Plus is an enhanced version of Stanford University's large-scale chest X-ray dataset, which adds alignment between structured annotations and natural language reports compared to earlier versions. It is considered one of the highest-quality publicly available RRG datasets. However, the "configuration gap" between dataset release and usable baselines is a common phenomenon in medical AI — high-quality datasets often lack standardized training scripts, hyperparameter settings, and evaluation pipelines, meaning that different papers claiming to evaluate on the same dataset may not be operating under comparable conditions. Commonly used evaluation metrics include natural language generation metrics such as BLEU, ROUGE, and CIDEr, as well as more specialized measures of clinical accuracy like F1 RadGraph and BERTScore. Establishing a unified benchmark means fixing these evaluation details so that performance numbers are truly comparable across methods.

MambaXray-PRB: Three-Stage Training and Multimodal Chain-of-Thought

Beyond benchmark construction, the paper's core contribution is rethinking the X-ray RRG task within the paradigm of large models and proposing the MambaXray-PRB framework. The name's PRB corresponds to three key concepts: Pre-training, Reasoning, and Benchmarking.

The framework improves both report generation performance and model interpretability through multi-stage large model pre-training and multimodal Chain-of-Thought (CoT) reasoning. The entire pipeline is divided into three consecutive stages:

Self-Supervised Autoregressive Modeling

The first stage pre-trains the model in a self-supervised autoregressive manner, enabling the model to learn low-level representations of X-ray images without requiring large amounts of annotations. This lays the foundation for subsequent cross-modal alignment.

Self-Supervised Learning is a training paradigm that extracts supervisory signals from the data itself without manual annotation. In medical imaging, where expert annotation costs are extremely high, self-supervised methods have significant practical value. Autoregressive Modeling is one common strategy — the model is trained to sequentially predict the next element in a sequence (which could be pixel patches, image patches, or text tokens), thereby being forced to learn the intrinsic statistical patterns and structural features of the data. For X-ray images, this process helps the model capture low-level visual representations such as lung field textures, bone contours, and lesion boundaries, providing a richer visual feature foundation for cross-modal alignment in subsequent stages.

X-Ray–Report Contrastive Learning

The second stage introduces contrastive learning between X-ray images and report text, bringing the representations of matched image-text pairs closer together while pushing mismatched pairs apart. This step aims to establish precise mappings between imaging features and diagnostic language.

The core idea of Contrastive Learning is to bring "positive sample pairs" closer together in representation space while pushing "negative sample pairs" apart, forming meaningful semantic clusters in the embedding space. In cross-modal settings, positive pairs are typically truly matched image-text combinations, while negative pairs are randomly assembled mismatches. Representative methods like CLIP have validated this approach on general vision-language tasks. The particular challenge in radiology is that visual differences between X-ray images are often far more subtle than in natural images — for example, mild pneumonia and a normal lung field can look extremely similar — so the model must learn to capture fine-grained differences at the medical semantic level during contrastive training, rather than relying solely on coarse-grained visual features.

Post-Training Optimization for Reasoning and Report Generation

The third stage performs post-training optimization for reasoning and report generation tasks. Through multimodal Chain-of-Thought, the model no longer simply "describes what it sees" but instead demonstrates a reasoning chain, improving both the quality and interpretability of generated reports — which is particularly important in medical settings, where clinicians need to understand the basis for AI conclusions.

Multimodal Chain-of-Thought (Multimodal CoT) extends chain-of-thought reasoning to joint vision-language scenarios. Traditional Chain-of-Thought Prompting, introduced by Google Brain in 2022, involves having language models generate intermediate reasoning steps before giving a final answer, significantly improving accuracy on complex reasoning tasks. The multimodal version requires the model to simultaneously reference visual evidence and textual context for step-by-step inference — for example, first identifying abnormal regions in the image, then combining clinical knowledge to infer possible pathological types, and finally generating a standardized report. This approach gives the model's decision-making process a degree of transparency, which has practical significance for both regulatory compliance and building clinical trust in medical AI.

Experimental Validation and Open-Source Value

The research team conducted extensive experiments on three datasets — IU X-ray, MIMIC-CXR, and CheXpert Plus — to validate the effectiveness of MambaXray-PRB on radiology report generation tasks. Cross-validation across multiple datasets strengthens the persuasiveness of the results, suggesting the framework has a degree of generalizability rather than simply overfitting to a single dataset.

Notably, the paper's source code has been open-sourced on GitHub (Event-AHU/Medical_Image_Analysis). Open-sourcing not only makes experimental results reproducible but also lowers the barrier for subsequent researchers to build on this work, aligning with the paper's emphasis on "standardized comparison."

Summary

This work has dual significance: on one hand, it fills the missing benchmark evaluation for CheXpert Plus — an important medical dataset — addressing the field's pain point around fair comparison; on the other hand, MambaXray-PRB combines the Mamba architecture, multi-stage pre-training, and multimodal Chain-of-Thought to provide a new approach to X-ray report generation that balances both performance and interpretability. For researchers and engineers focused on deploying medical AI in practice, this is an open-source project well worth following.