SGLang v0.5.21 Released: Performance, Scheduling, and New Model Support Across the Board

SGLang, one of the most widely followed LLM inference frameworks, has released version v0.5.21. This update brings together 779 PRs from 227 contributors, covering core areas including speculative decoding, parallel and disaggregated deployment, quantization, and caching systems — alongside support for a new batch of LLMs and diffusion models. For teams focused on inference performance and deployment efficiency, this release contains quite a few changes worth a closer look.

PD Role Hot-Switching: More Flexible Disaggregated Deployment

One of the most notable capabilities in this release is the ability to dynamically switch Prefill/Decode (PD) instance roles at runtime without restarting the service (#28403). In a disaggregated architecture (Prefill-Decode Disaggregation), the prefill and decode stages are typically assigned to separate instances to improve throughput — but fixed role assignments struggle to handle dynamically shifting workloads. Runtime role switching means a cluster can adjust resource allocation based on real-time request patterns, better matching compute-intensive prefill and memory-bandwidth-intensive decode to actual traffic.

There's a long list of additional PD improvements as well. For example, using request-owned speculative KV reduces decode startup latency — in a DeepSeek^Ad-V4-Pro 1P1D test on MI355X at concurrency 256, average TTFT (Time To First Token) dropped from 19.4 seconds to 15.1 seconds (#38978). In a Kimi Linear test on 8x B200 at concurrency 8, cached-prefix throughput improved by approximately 10.9x (#40376). These numbers reflect significant engineering investment in scheduling and KV transfer pipelines for disaggregated deployments.

Prefill-Decode Disaggregation (PD disaggregation) is an important optimization approach in LLM inference. In traditional deployments, a single GPU instance handles both the prefill stage (processing the input prompt — compute-intensive) and the decode stage (generating tokens one by one — memory-bandwidth-intensive). Since these two stages have very different hardware requirements, running them together leads to uneven GPU utilization. PD disaggregation splits the two stages across separate instances: prefill instances handle high-parallelism matrix operations, while decode instances focus on frequent KV cache reads and writes — allowing each to be independently optimized for hardware configuration and batching strategy.

TTFT (Time To First Token) is a core metric for user-perceived latency, measuring the time from sending a request to receiving the first output token. It is primarily influenced by the prefill stage. ITL (Inter-Token Latency) measures the average or percentile time to generate each new token during the decode stage, directly affecting the smoothness of streaming output. Together, these two metrics define the overall user experience of an inference service.

Rust Prefix Cache Now the Default Implementation

The caching system is a critical factor in inference framework performance. This release switches the core prefix cache implementation to Rust and makes it the default (#39627). The Rust-based TreeCore offers advantages over the previous implementation in memory management and concurrency safety. Replacing the default backend reflects that this implementation has reached production-ready maturity.

The unified Radix Cache also gains an optional Agentic-Aware tail-optimized LRU eviction policy. In a DeepSeek-V4-Pro FP4 test on 4x B300 using agentic traces at concurrency 32, p99 ITL (inter-token latency) dropped by 43.9% (#34012). Targeted optimization for the long-tail cache access patterns characteristic of agentic workloads signals that the framework is actively adapting to the new load profiles of agent-based applications.

Prefix Cache (also known as KV cache reuse) is an important inference acceleration technique. When a large model processes each token, it generates Key-Value tensors that can be reused across requests: if multiple requests share the same prefix (such as a common system prompt or conversation history), the KV computations for that portion can be read directly from cache rather than recomputed, significantly reducing TTFT and improving throughput. Radix Tree is the core data structure for efficient prefix matching — it stores prefix paths of varying lengths in a tree structure, enabling fast O(n) lookups and shared node reuse.

LRU (Least Recently Used) is the most common cache eviction policy, prioritizing the removal of the least recently accessed cache blocks. The "Agentic-Aware tail-optimized LRU" designed for agentic workloads improves on this — agentic applications tend to frequently append new content at the end of long trajectories, and a standard LRU may prematurely evict tail cache blocks that are about to be accessed again. The specialized policy addresses this by recognizing such access patterns to improve cache hit rates.

New Classification and Scoring APIs

Beyond inference performance, v0.5.21 expands the use cases for LLMs and VLMs at the interface level. The newly introduced Decisions API (/v1/decisions) turns large models into low-latency classifiers and scorers (#41208), while the Score API (/v1/score) supports scoring all candidates in a single request (#38965, #41188).

The significance of these APIs lies in wrapping generative model capabilities into more structured, controllable discriminative task outputs. For scenarios that require LLMs to handle intent recognition, content moderation, or candidate ranking, dedicated endpoints eliminate the uncertainty of parsing free-form text output while delivering more predictable latency.

Speculative Decoding and Model Acceleration

Speculative Decoding receives a dense set of updates in this release, including compatibility with Pipeline Parallelism and EAGLE/MTP (#30775), XQA backend support (#32269), and DFlash draft schemes for various models. Among these, LFM2-VL achieves 1.66x to 2.56x speedup at batch size 1 via DSpark speculative decoding (#40651).

Per-model speedups are also substantial: DeepSeek-V4.1 achieves a 22% improvement in time-to-first-token on long prompts (#40352); Kimi K3 sees a 20.6% increase in prefill throughput in PD serving (#40045); and GLM-5.3-Flash now runs on AMD MI355X with support for FP8/MXFP4 MoE and MTP speculative decoding (#38546 and related). Docker images for multiple hardware platforms (NVIDIA CUDA 13, AMD MI35x/MI30x, Intel GPU/CPU) are also provided.

Speculative Decoding is a technique that accelerates autoregressive generation through a "small model drafts, large model verifies" approach. The core idea: a small draft model quickly generates several candidate tokens, which the target large model then verifies in a single parallel pass. If the draft tokens are accepted, multiple decoding steps are effectively collapsed into one forward pass, significantly improving throughput and speed. EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) and MTP (Multi-Token Prediction) are two mainstream draft generation schemes: EAGLE trains a lightweight draft head using the target model's feature layers, while MTP trains the model to simultaneously predict multiple future tokens, enabling speculative inference without an additional model. Compatibility improvements for combining both with Pipeline Parallelism allow speculative decoding to run reliably in multi-GPU distributed settings.

New Model Support

This release adds out-of-the-box support for a new set of models. On the LLM/VLM side: DeepSeek-V4.1 Flash, GigaChat 3.5, IQuest-Q1, MiMo-V2.6 series, and Ling-3.0-flash-VL. For diffusion models: DiffusionGemma, Qwen-Image 2.1, Anima Base v1.0, Ming-Image 0.1, and FLUX 3 Action. Additionally, MiniMax-H3 can run in ComfyUI via SGLang Diffusion (#35990), further bridging integration with existing ecosystem tools.

How to Upgrade

To upgrade to this version:

uv pip install --prerelease=allow sglang==0.5.21

Or pull the Docker image for your platform — for example, NVIDIA (CUDA 13):

docker pull lmsysorg/sglang:v0.5.21

Summary

Overall, SGLang v0.5.21 is not a single breakthrough but a systematic refinement across the full inference serving stack: from PD role hot-switching and Rust prefix cache, to numerical precision correctness fixes and improved scheduling strategies, to discriminative APIs and multi-hardware support. For inference deployments that demand high throughput and low latency — especially under complex workloads like agentic scenarios — the engineering value of this release is considerable.