NVIDIA NV-Reason-CT: A Native 3D Vision-Language Model for CT Scan Interpretation

NVIDIA has open-sourced a medical imaging model called NV-Reason-CT, capable of reading CT scans in native 3D, reasoning step-by-step, and generating radiology reports. Unlike most vision-language models that treat CT scans as stacks of flat images, NV-Reason-CT preserves complete three-dimensional spatial information from the start — addressing a long-overlooked pain point in medical imaging AI.

Why Existing Vision-Language Models Struggle with CT Scans

A single CT scan contains hundreds of slices, forming a complete 3D record of the chest or abdomen. However, mainstream vision-language models are designed for flat images. When faced with CT data, they can only sample a handful of slices or compress the entire volume into a few tokens. Either way, critical spatial detail is lost before reasoning even begins.

This information loss is especially costly in medical contexts — a tiny calcified nodule or the 3D positional relationship of a lesion can determine a diagnosis. The NVIDIA team therefore chose a different technical path: processing volumetric data directly with a native 3D visual encoder.

Native 3D Architecture: End-to-End Training from Encoder to Language Model

At the core of NV-Reason-CT is a native 3D visual encoder whose output is fed into the QWEN 3.5 language model, with both components trained end-to-end jointly. This design enables the model to produce structured radiology reports, answer questions, and demonstrate reasoning over chest and abdominal CT scans.

During preprocessing, the team uses a processor called MoniTransforms to load scans, resample them, and automatically crop the chest or abdominal region. The visual encoder then divides the volumetric data into 3D grid patches, generating 13,824 visual tokens in total. Each token is passed to the language model along with its position tag in 3D space.

This approach — where every token carries 3D coordinates — is key to preserving the model's spatial reasoning capability. Compared to flattening 3D data into 2D, it allows the language model to understand the spatial relationships of anatomical structures during inference.

The idea of 3D patch partitioning is inspired by how Vision Transformers (ViT) handle images: dividing the input space into regular blocks, representing each with a vector, and feeding them into an attention mechanism. For 2D images, a patch is a rectangular H×W block of pixels; for volumetric data, it becomes an H×W×D cube, often called a "tube" or "cuboid." The number 13,824 means the model must simultaneously process roughly 14,000 visual tokens in its context window — a significant stress test of the language model's long-sequence capabilities. The additional 3D positional encoding (typically implemented as sinusoidal functions or learnable embeddings) gives each token its coordinates along the x, y, and z axes, allowing the language model to perceive spatial proximity between anatomical structures as attention propagates across layers, rather than treating all tokens as an unordered set.

Training Data and Reinforcement Learning Fine-Tuning

The model's training scale is substantial: approximately 550,000 question-answer samples derived from around 70,000 CT volumes, with reasoning guided by actual radiologist interpretations.

Training proceeds in two stages: supervised fine-tuning followed by reinforcement learning optimization — with reward signals directly targeting whether imaging findings are correctly identified. This diagnostically-oriented reinforcement learning means the model's optimization objective is closely aligned with clinical priorities, rather than simply chasing linguistic fluency.

The reinforcement learning fine-tuning mentioned here typically refers to RLHF (Reinforcement Learning from Human Feedback) or its variants in the large language model domain, but the source of reward signals differs in the medical imaging context: instead of relying on human preference scores, the structured "imaging findings" field in radiology reports serves as a quantifiable reference answer, and the reward is computed from the match between model output and radiologist records. This "Outcome-Based RL" approach has the advantage of forcing the model to optimize whether the final diagnostic conclusion is correct, rather than whether the language is fluent or the length is appropriate. Recent research on reasoning models (such as DeepSeek^Ad-R1) has shown that this type of reinforcement learning with verifiable outcomes can significantly improve the quality of logical reasoning chains. Medical imaging report generation similarly offers verifiable outcomes, giving this strategy a strong methodological basis for transfer to this domain.

Inference Performance and Benchmark Results

In the official demo, users can ask the model to perform a complete analysis of a chest scan. The model walks through anatomical regions step by step, noting findings such as emphysema, tiny calcified nodules, and hiatal hernias, before arriving at a set of definitive conclusions.

The model supports flexible interaction modes: the "thinking" process can be disabled for quick yes/no answers, or users can continue asking follow-up questions about the same scan. On the CT-RATE Abnormality benchmark, NV-Reason-CT achieved the highest macro F1 score among compared models using only a simple yes/no prompt, with no classification head required. This demonstrates that its anomaly detection capability does not stem from a specialized classification module, but from the language model's own reasoning.

CT-RATE is a publicly available large-scale chest CT dataset and benchmark containing paired radiology reports and multi-label abnormality annotations, commonly used to evaluate model performance on detecting conditions such as pulmonary nodules, pneumothorax, and emphysema. The macro F1 score assigns equal weight to each category before averaging, making it equally sensitive to rare abnormality classes — which aligns well with clinical practice, since missing a low-frequency but high-risk finding is typically more consequential than misdiagnosing a common one. NV-Reason-CT's ability to achieve the highest macro F1 without attaching a dedicated classification head to the output layer suggests that its understanding of "abnormalities" is fully internalized within the language model's generation probabilities. This points to strong zero-shot generalization potential and serves as a reference point for future transfer to other imaging modalities or pathology types.

Open-Source Ecosystem and Availability

NV-Reason-CT is now fully open: model weights are hosted on Hugging Face, code and fine-tuning scripts are published on GitHub alongside a companion skill, and the associated paper is available on arXiv. For teams looking to build applications or conduct research in medical imaging, this complete open-source package lowers the barrier to reproduction and further development.

From a technical standpoint, NV-Reason-CT represents a shift in multimodal models from "adapting to flat images" toward "natively understanding 3D medical data." It combines interpretable step-by-step reasoning with objectively validated accuracy on benchmarks, offering a compelling architectural paradigm for future medical AI research.