TileRT
An inference runtime tool focused on optimizing the decode phase of large model inference, with targeted optimizations for low-latency autoregressive generation scenarios. It can be used alongside mainstream frameworks such as vLLM to enable disaggregated inference deployment.
Timeline (last 90 days)
On 8 AMD Instinct MI355X accelerator cards, using the vLLM inference framework, the GLM-5.3 model achieved a single-user decoding speed of 469 tok/s.
The inference deployment adopts a disaggregated architecture, with the prefill phase handled by vLLM and the decode phase taken over by TileRT.
TileRT and vLLM are connected through the connector interface of vLLM V1.
The above performance data comes from a public demonstration by a single source, and independent reproduction and complete test details (such as batch configuration, quantization scheme, and sequence length) are not yet available.
All Facts (4)
On 8 AMD Instinct MI355X accelerator cards, using the vLLM inference framework, the GLM-5.3 model achieved a single-user decoding speed of 469 tok/s.
50%UnverifiedThe inference deployment adopts a disaggregated architecture, with the prefill phase handled by vLLM and the decode phase taken over by TileRT.
50%UnverifiedTileRT and vLLM are connected through the connector interface of vLLM V1.
50%UnverifiedThe above performance data comes from a public demonstration by a single source, and independent reproduction and complete test details (such as batch configuration, quantization scheme, and sequence length) are not yet available.
50%