AMD MI355X Beats B300 in Inference Performance: 24.4% Higher Throughput on 8 GPUs

The battle between AMD and NVIDIA in the AI inference market has reached a new turning point. According to SemiAnalysis's AgentX agentic coding benchmark, an 8-GPU AMD MI355X configuration serving the Kimi K3 model achieved 7,567 tok/s per GPU at an interactive latency of 50 tok/s/user—24.4% higher than HGX B300's 6,083 tok/s/GPU. Measured in tokens per dollar, the advantage widens to 87.4%. The results were jointly disclosed by @Kimi_Moonshot, @AMD, and @embeddedLLM, along with full details of the vLLM optimizations and AMD kernel implementations.

Benchmark Results: A Double Win on Throughput and Cost-Efficiency

AgentX is an agentic coding evaluation suite released by SemiAnalysis, with workload characteristics that closely mirror real coding-agent scenarios: long contexts, multi-turn tool calls, and frequent short outputs. Under this workload, the 8× MI355X not only leads in throughput but also opens a significant gap in output per unit cost.

The key data comparison is as follows:

• Throughput: MI355X 7,567 tok/s/GPU vs. B300 6,083 tok/s/GPU, a 24.4% lead
• Tokens per dollar: MI355X is 87.4% higher
• Test conditions: 50 tok/s/user interactive speed, Kimi K3 model, vLLM inference framework

The 87.4% cost-efficiency advantage does not come purely from peak hardware compute, but from software-hardware co-optimization. This also explains why AMD chose to work with teams like embeddedLLM that specialize in the inference stack, rather than relying solely on chip specifications.

The difference between tok/s/user and tok/s/GPU: The former measures the generation speed experienced by a single user—that is, the number of tokens per second each user receives during the decode phase; the latter measures aggregate throughput for the whole system or card. In batched inference, there is a trade-off between the two: increasing batch size can raise total throughput but lengthens per-request latency. Therefore, "achieving 7,567 tok/s/GPU at 50 tok/s/user" means maintaining high aggregate throughput while ensuring smooth interactivity—a far better reflection of production readiness than a benchmark that simply maximizes batch size.

PagedAttention: vLLM's core memory management mechanism, inspired by the virtual memory paging of operating systems. It divides the KV Cache into fixed-size blocks allocated on demand, eliminating the memory waste caused by reserving the maximum sequence length and significantly increasing batch size. Its memory layout and block scheduling strategy depend heavily on GPU architecture characteristics, and migrating to the ROCm platform often requires re-tuning.

How the Gap Was Closed: vLLM Optimizations and AMD Kernels

According to the disclosure, the performance gains are mainly attributable to three areas of work: scheduling and memory management optimizations at the vLLM level, kernel implementations customized for AMD architecture, and detailed hardware-level tuning.

As a mainstream open-source inference engine, vLLM's default path has long been optimized around NVIDIA GPUs. Unleashing MI355X's potential requires re-examining PagedAttention's memory layout, continuous batching strategy, and the expert routing overhead of MoE models. The contribution of AMD kernels lies in filling efficiency gaps in the ROCm ecosystem for specific operators (such as attention and normalization).

The significance of this combined effort extends beyond a single benchmark run: it provides a reusable pattern for bottleneck identification, enabling users to squeeze more performance out of vLLM on their own GPUs.

ROCm and kernel implementations: ROCm is AMD's open-source GPU computing software stack, positioned as a counterpart to CUDA. A kernel refers to a parallel compute function executed on the GPU; if operators such as attention, normalization, and MoE routing are not hand-optimized for a specific architecture, performance can fall far short of peak. One focus of the AMD–embeddedLLM collaboration is to fill in efficient implementations of these high-frequency operators for MI355X's matrix cores and memory hierarchy, reducing inefficient memory access and synchronization overhead in the framework's default paths.

Serving challenges of MoE models: Mixture-of-Experts models activate only a subset of experts per forward pass, resulting in large memory footprint but relatively sparse computation. The main bottleneck during serving is the irregular memory access and load imbalance caused by expert routing, which requires coordinated handling by the scheduler and kernels—otherwise GPU utilization drops noticeably. Kimi K3 belongs to this architecture, which is why the benchmark results are highly sensitive to inference-stack optimization.

Signals for the Inference Market

NVIDIA's advantage on the inference side has long been built on the CUDA ecosystem and engineering maturity rather than raw hardware specifications alone. MI355X's ability to match or even surpass B300 on real agentic workloads shows that AMD's weakness is shifting from "the hardware isn't good enough" to "the software needs refinement"—and the latter can be closed with investment.

For buyers, a nearly twofold gap in tokens per dollar is enough to change TCO calculations. For developers, the combination of ROCm and vLLM is beginning to reach production readiness. Of course, a single benchmark cannot represent all workloads; training, fine-tuning, and ultra-long-context scenarios still require independent validation.

Variables Still to Watch

This data was jointly released by AMD and its partners, and independent third-party reproduction has not yet been made public. Although AgentX closely mirrors coding-agent scenarios, model serving performance depends heavily on specific kernel versions and compilation options. There may also be unreleased software optimization headroom on the B300 side.

What is certain is that competition in inference hardware has moved from "whose chip is faster" to "whose stack is more complete"—which is good for the industry as a whole.

One-line Summary

AMD MI355X outperforms B300 by 24.4% in throughput on an agentic coding benchmark, with 87.4% more tokens per dollar.

Paragraph Summary

SemiAnalysis's AgentX agentic coding benchmark shows that an 8-GPU AMD MI355X configuration serving the Kimi K3 model achieves 7,567 tok/s per GPU at 50 tok/s/user interactive latency—24.4% higher than HGX B300's 6,083 tok/s/GPU—with the advantage widening to 87.4% in tokens per dollar. The gains mainly come from vLLM scheduling and memory management optimizations, kernels customized for AMD architecture, and detailed hardware tuning, reflecting software-hardware co-optimization rather than a pure compute lead. The article argues this marks a shift in AMD's weakness from hardware to a catchable software ecosystem, enough to change buyers' TCO calculations—but the data was jointly released by vendors and awaits independent reproduction, while training and ultra-long-context scenarios still need separate validation.