AMD Teams Up with vLLM to Launch d3-ultra: A New Breakthrough in Open-Source Multimodal Decision-Making

AMD and the vLLM Semantic Router team have collaborated to launch the d3-ultra model, bringing new possibilities to the field of open-source multimodal decision-making. The model supports per-answer probability outputs, visual grounding, and video tracking, and is trained and inferred entirely on AMD Instinct GPUs. More importantly, the model weights are open-sourced under the Apache-2.0 license on Hugging Face, meaning developers can freely use, modify, and distribute them.

Core Capabilities of d3-ultra

Based on publicly available information, d3-ultra focuses on "open multimodal decision-making." It is not just a vision-language model; it emphasizes providing interpretable outputs during decision-making—each answer comes with a probability value. This design makes the model more practical in scenarios that require weighing multiple options, such as medical image analysis, autonomous driving perception decisions, or complex document understanding.

Visual grounding capability means the model can not only answer "what is in the image" but also point out the specific location of the target in the image. Video tracking further extends the capability to temporal data, enabling continuous tracking of objects in videos. Combined, these features bring d3-ultra closer to a complete perception-decision system rather than a simple question-answering model.

Training and Inference Entirely on AMD Instinct GPUs

A major highlight of this model is that both training and inference are completed on AMD Instinct GPUs. For a long time, NVIDIA GPUs have dominated the AI training field, with AMD's ROCm software stack and Instinct accelerators playing catch-up. The emergence of d3-ultra shows that AMD hardware can now support the complete training pipeline for multimodal models, which is a positive signal for enterprises and research institutions seeking GPU supply diversification.

The involvement of the vLLM Semantic Router team is also noteworthy. vLLM itself is an important open-source project in the efficient inference space, while Semantic Router involves routing decisions based on semantics. The combination may imply that d3-ultra has specialized optimizations for inference efficiency and service deployment, though specific performance data has not yet been disclosed in the tweet.

ROCm (Radeon Open Compute) is AMD's open-source software stack comparable to CUDA, including compilers, runtimes, math libraries, and communication libraries, responsible for mapping computation graphs from frameworks like PyTorch and TensorFlow onto Instinct accelerators. For a long time, ecosystem maturity and operator coverage have been the main bottlenecks for AMD in catching up with NVIDIA, especially for multimodal training involving large-scale distributed communication, mixed precision, and attention operators. d3-ultra claims that the entire training and inference pipeline is completed on Instinct, and its significance lies in validating the usability of ROCm in these areas, rather than merely running single-card inference.

AMD Instinct is AMD's data center GPU product line (such as the MI200 and MI300 series), using the CDNA architecture, emphasizing FP64/FP32 and matrix operation throughput, and supporting multi-card interconnect via Infinity Fabric. Unlike consumer-grade Radeon, Instinct is optimized for HPC and AI training scenarios and is typically deployed as complete systems or clusters. When evaluating such models, beyond checking whether they "can run on AMD," one should also consider multi-card scaling efficiency, memory bandwidth utilization, and operator compatibility with mainstream frameworks.

Significance of the Apache-2.0 Open-Source License

The model weights are released under the Apache-2.0 license on Hugging Face, which is a fairly permissive open-source license. Commercial use, modification, and redistribution are all permitted, requiring only retention of copyright and license notices. For enterprises and developers looking to build upon multimodal decision-making, this reduces legal risks and technical barriers.

The choice of Apache-2.0 also suggests the intent of AMD and the vLLM team to promote ecosystem building—attracting more developers to innovate on the AMD platform through open weights, thereby indirectly enhancing the appeal of the ROCm ecosystem.

It is important to distinguish "open weights" from "fully open source": releasing weights under Apache-2.0 on Hugging Face means the license terms cover the model parameter files and accompanying code, but training data, data cleaning pipelines, and the complete training recipe are typically not disclosed. Developers can freely use commercially and redistribute the weights, but may not be able to reproduce the model's training process. Additionally, Apache-2.0 does not include usage restrictions beyond patent retaliation clauses, nor does it require derivative works to be released under the same license, which is fundamentally different from GPL-type copyleft licenses and therefore more friendly for enterprise secondary development and closed-source integration.

In model licensing practice, Apache-2.0 is among the least restrictive tiers, contrasting with the custom community licenses of the Llama series and the non-commercial terms of some models. Choosing this license typically aims to lower adoption barriers and accelerate ecosystem diffusion, but it also means the publisher gives up the ability to constrain downstream usage.

A New Variable in the Multimodal Decision-Making Race

Multimodal models are moving from "describing the world" to "making decisions in the world." Per-answer probability output is a key feature in this transition: it allows model outputs to be quantitatively evaluated by downstream systems rather than relying solely on human judgment. Visual grounding and video tracking further broaden the application boundaries.

The release of d3-ultra provides the open-source community with a new base option, especially within the AMD hardware ecosystem. However, to judge its actual impact, we need to observe subsequent benchmark results, community adoption, and inference performance. Currently, public information is mainly feature announcements; detailed model size, training data composition, and evaluation data have not yet been provided in the original tweet.

For readers interested in AI infrastructure diversification and open-source multimodal models, d3-ultra is worth following.

Per-answer probability refers to the model providing a normalized confidence score for each candidate answer, rather than just generating a piece of text. This enables downstream systems to perform threshold filtering, multi-answer ranking, or uncertainty estimation, which is particularly important in high-risk scenarios such as healthcare and autonomous driving. Implementation typically involves adding a classification head at the output layer, scoring candidate answers by likelihood, or applying softmax normalization to candidate logits. It should be noted that such probabilities are often the model's internal relative confidence and may not be well-calibrated; using them directly as true probabilities can lead to overconfidence issues.

Visual grounding requires the model to map textual references to image regions, with common output forms being bounding box coordinates or pointing segmentation masks, and evaluation metrics mostly using IoU or pointing accuracy. Video tracking introduces temporal consistency on this basis, requiring maintaining target identity across frames, involving data association and occlusion handling. Together, they move the model from static description to structured output with spatiotemporal localization, but also impose higher demands on annotated data and inference latency.

One-line Summary

AMD and the vLLM team open-source the d3-ultra multimodal decision model, with the entire pipeline trained and inferred on Instinct GPUs.

Paragraph Summary

AMD and the vLLM Semantic Router team have collaborated to launch the d3-ultra open-source multimodal decision model, supporting per-answer probability outputs, visual grounding, and video tracking, with training and inference entirely on AMD Instinct GPUs. The weights are released under the Apache-2.0 license on Hugging Face, allowing commercial use and redistribution. The article argues that the model's value lies in advancing multimodal capabilities from "describing the world" to "making decisions," while validating the feasibility of AMD hardware and the ROCm software stack supporting multimodal training pipelines. The involvement of the vLLM team suggests possible specialized optimizations in inference efficiency and semantic routing, but model size, training data composition, and benchmark evaluation data have not yet been disclosed, and actual impact remains to be seen.