GLM-5.3 Hits 469 tok/s on 8×MI355X: A Deep Dive into vLLM + TileRT Disaggregated Inference

An Engineering Milestone in Single-User Decode Speed

Recently, TileRT (@TileRT_AI) and the AMD AI team (@AIatAMD) showcased a notable inference performance result: running GLM-5.3 on 8 AMD Instinct MI355X accelerators via the vLLM inference framework, they achieved 469 tok/s single-user decode speed. The benchmark was conducted in SemiAnalysis's AgentX evaluation environment.

What makes this number stand out is what it actually measures — single-user decode, meaning the number of tokens generated per second for a single request, not aggregate throughput inflated by large batch concurrency. Single-user latency is the core metric for interactive applications like conversational AI and agent-based reasoning, and it's significantly harder to optimize than batch throughput.

Disaggregated Architecture: Prefill and Decode, Each in Its Lane

The technical centerpiece of this achievement is a disaggregated inference deployment. According to the official description, the full inference pipeline is split into two distinct stages, each handled by a different component:

• Prefill is handled by vLLM. This stage involves the model processing the input prompt in parallel — a compute-intensive task where vLLM's mature scheduling and batching capabilities shine.
• Decode is handled by TileRT. This is the autoregressive, token-by-token generation phase, which is highly latency-sensitive. TileRT is specifically optimized for this latency-critical path.

The two components are connected via vLLM V1's connector interface, allowing TileRT to integrate seamlessly into vLLM's workflow. This principle of "let the specialist handle its domain" is one of the most important directions in LLM inference optimization today.

Why Separate Prefill and Decode?

Prefill and decode have fundamentally different computational characteristics. Prefill processes the entire input sequence in one parallel pass, keeping GPU compute utilization high. Decode, on the other hand, generates only one token per step and is bottlenecked by memory bandwidth and kernel launch overhead — it's hard to fully saturate compute resources this way. Mixing both in the same scheduler tends to create interference that degrades both.

Disaggregated deployment lets each stage run under the configuration best suited to it — prefill optimized for compute throughput, decode optimized for low latency — yielding better end-to-end performance overall. This architectural choice is precisely what makes the 469 tok/s single-user decode figure achievable.

---

From a systems perspective, the prefill stage is essentially one large matrix multiplication (with higher parallelism for longer inputs), where GPU compute units (CUDA Cores / Shaders) can be fully saturated — a compute-bound problem. The decode stage, by contrast, reads the full KV Cache once per step and writes back a tiny amount of data, making memory bandwidth the bottleneck rather than compute — a memory-bound problem. These two fundamentally different bottlenecks call for different kernel implementations, scheduling granularities, and even hardware configurations. Traditional monolithic inference servers must compromise between the two modes. Disaggregated serving physically separates these stages and hands each to a runtime tailored for it — effectively implementing "pipeline specialization" within the same hardware cluster, allowing each stage to operate at its own optimal point.

Hardware Perspective: MI355X and the Open Inference Ecosystem

This benchmark ran on 8 AMD MI355X cards, signaling AMD's continued push in high-end AI acceleration hardware. For a long time, the performance narrative in large model inference was dominated by a single vendor. AMD's collaboration with open-source and third-party toolchains like vLLM and TileRT to demonstrate competitive inference numbers signals that hardware choices for inference deployment are becoming more diverse.

vLLM, one of the most widely adopted open-source inference frameworks today, provides a connector interface in its V1 architecture that leaves room for third-party optimization solutions like TileRT to plug in. This open, plugin-style design allows different teams in the ecosystem to stack their own optimizations on top of a unified framework — without each having to rebuild the entire stack from scratch.

---

AMD Instinct MI355X is AMD's data center AI accelerator based on the CDNA 4 architecture. Compared to its predecessor, the MI300X, it delivers improvements in both memory capacity and bandwidth, featuring HBM3E memory with over 6 TB/s of total bandwidth per card. For memory-bound workloads like the decode stage, higher bandwidth translates directly into higher token generation rates. The MI355X also supports FP8 precision inference, which can further reduce memory footprint and bandwidth pressure while preserving model accuracy. On the software side, AMD's ROCm platform now offers a programming interface largely aligned with CUDA, and support for ROCm backends in mainstream frameworks like vLLM continues to mature — enabling third-party optimization components like TileRT to integrate into AMD's hardware stack without requiring major changes to higher-level code.

What These Numbers Actually Mean

For developers and engineering teams focused on LLM deployment, this announcement carries several key implications:

• Single-user decode speed is becoming a critical battlefield in inference optimization, directly determining usability in interactive scenarios like agents and real-time conversation;
• Disaggregated inference architecture (decoupling prefill and decode) has demonstrated clear practical value and deserves evaluation in production deployments;
• The combination of AMD hardware + open-source frameworks + specialized optimization components opens up more viable paths for inference deployments.

It's worth noting that these numbers come from a single public demonstration and currently lack independent reproduction or full test details (e.g., specific batch configurations, quantization schemes, sequence lengths). Readers should assess them critically in the context of their own use cases and wait for further technical disclosures.

---

SemiAnalysis's AgentX benchmark is an inference evaluation suite designed specifically for AI agent scenarios, with a core focus on single-user decode latency rather than the maximum throughput-oriented batch testing that traditional benchmarks favor. This shift in perspective reflects a broader industry reassessment of inference evaluation standards: in real-world agent workflows, a single reasoning chain often requires the model to complete a response within hundreds of milliseconds, making low latency more directly decisive for task quality than high throughput. At 469 tok/s, generating 500 tokens takes just over one second — a speed that carries strong practical significance for agentic tasks requiring rapid, iterative reasoning steps.

Summary

GLM-5.3's 469 tok/s single-user decode on 8×MI355X is the result of deliberate engineering: disaggregating prefill and decode, and assigning vLLM and TileRT to their respective roles. It's both a concrete performance demonstration and a reflection of how large model inference is evolving toward more granular specialization and a more open ecosystem. For teams wrestling with high-concurrency or low-latency inference challenges, this disaggregated approach may well be a reference architecture worth studying closely.