SGLang v0.5.17 Deep Dive: Kimi K3 Day-0 Support, Rust Frontend, and 5x Faster Model Loading

The open-source LLM inference engine SGLang has released v0.5.17 — a massive update comprising 582 PRs from 194 contributors. The highlights include day-0 support for cutting-edge models like Kimi K3 and MiniMax-H3, along with deep performance optimizations across model loading, MoE parallelism, speculative decoding, and more. This article breaks down the core changes and analyzes their technical significance.

Day-0 Model Support: Kimi K3 and MiniMax-H3

Kimi K3: A 2.8 Trillion Parameter Multimodal LatentMoE Model

The most eye-catching addition in this release is native support for Kimi K3 — a 2.8T-parameter multimodal LatentMoE model with 896 experts, top-16 routing, and routing computations performed in a 3584-dimensional latent space. Its architecture is remarkably distinctive: a 1 million token context length, 69 KDA linear attention layers interleaved with 24 MLA layers, a MoonViT3d vision tower, and released as native MXFP4 checkpoints.

SGLang provides full serving capability from day one, including DCP (DeepSeek Context Parallelism), DSpark speculative decoding, pipeline parallelism with chunked prefill paired with TP decoding, KDA-aware prefix caching, DCP-based HiCache L2 caching, and the ability to run LoRA on quantized weights. Support has been validated on both NVIDIA GB300 and AMD MI35x, demonstrating SGLang's depth of adaptation across heterogeneous hardware.

MiniMax-H3: Unified Video and Audio Generation

Another day-0 addition is MiniMax-H3 — a video generation model capable of simultaneously generating video and synchronized stereo audio tracks within a single request. It's served via SGLang-Diffusion and covers three publicly available task modes: text-to-video-plus-audio (t2va), first-and-last-frame conditioned generation (fl2va), and image/video/audio reference conditioned generation (ref2va, which also covers video-to-video conversion).

Validation spans from B200 and H100 to AMD MI300X/MI355X, and even dual RTX 5090 setups via layer-by-layer offloading — showing SGLang's consideration for users across different compute budgets. This release also adds support for EmbeddingGemma, LFM2.5 embedding models, and MiniMax-M3-NVFP4.

Rust Frontend Serving Layer: Addressing the Python GIL Bottleneck

A direction worth watching long-term is the introduction of a Rust serving layer. SGLang has begun migrating the first half of the server pipeline — everything from the network entry point to handing tokenized requests off to the GPU scheduler — from Python to a multi-threaded Rust implementation.

This native Rust serving layer includes a tokenization manager, request ingress validation and response handling, an OpenAI-compatible API server, and PD disaggregation (Prefill-Decode disaggregation) support, delivered as pre-built release artifacts. In high-concurrency scenarios, Python's GIL and scheduling overhead have always been a hidden bottleneck in inference systems. The introduction of Rust signals SGLang's intent to fundamentally reduce host-side overhead. Related work in this release also specifically fixes TCP-layer-induced TTFT (time-to-first-token) stalls, reflecting a serious focus on real-world production latency.

Major Inference Performance Improvements

5.6x Faster Model Loading

This is likely the improvement most immediately felt in day-to-day operations. Previously, oversized or non-contiguous CPU weight views caused pathological host-to-device (H2D) transfers — for DeepSeek-V4-Pro under TP8 configuration, the H2D phase on some ranks took anywhere from 27 to 32 minutes.

By copying these views into contiguous storage before the H2D transfer, full model load time dropped from roughly 35 minutes to 6 minutes 20 seconds (5.6x speedup); BF16 loading for GPT-OSS-20B went from 545 seconds to 70 seconds (7.8x), and Qwen3.5-397B saw 1.93x to 2.3x improvements. This optimization is enabled via the environment variable SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D and is off by default.

Fast Engine Recovery: Weight Caching Daemon

The cost of restarting large models is equally staggering — Qwen3-235B FP8 on 4 GPUs takes roughly 6.5 minutes to restart, due to weights needing to be reloaded from storage and CUDA graphs needing to be recaptured. This release introduces a weight caching daemon that retains weights per GPU, allowing a restarting engine to recover from cache rather than storage, significantly shrinking the failure recovery window.

DWDP Parallelism: 1.92x MoE Prefill Throughput Improvement

SGLang introduces a new prefill parallelism strategy called DWDP (Distributed Weight Data Parallelism). It prefetches peer expert weights via NVLink P2P and computes all experts locally, thereby eliminating the all-to-all token dispatch overhead inherent in EP (Expert Parallelism).

In pure prefill benchmarks running gpt-oss-120b on 4x B200, DWDP4 achieves a 1.92x speedup over DEP4 at MNT 32K / ISL 32K configuration, and delivers 506K vs. 329K tok/s (1.54x) throughput improvement under saturated conditions (CONC=128, ISL=8K). The authors note this feature is still in early development and can be enabled via --dwdp-size.

Memory Savings and Communication Optimizations

Significant Memory Reduction on AMD Platforms

On AMD platforms, by removing unnecessary expert padding, FP4 MoE model weights for DeepSeek-V4 on MI355X dropped from 159.07 GB to 112.36 GB. By incorporating the HIP compression state pool into memory_saver's KV_CACHE region, co-located reinforcement learning tasks can reclaim that memory, reducing per-GPU usage during training from ~143 GiB to 87 GiB.

Pluggable Communication Backends and the Helix Approach

On the communication front, the DeepSeek-MLA decode context parallelism path gains pluggable communication backends: a2a exchanges packed attention outputs and fp32 LSE via a single NCCL collective per layer; fi_a2a on GB200 delegates cross-rank exchanges to FlashInfer MNNVL kernels; and --dcp-replicate-q-proj skips the per-layer Q-head dimension all-gather by projecting full-head Q locally. This mechanism is what the community refers to as the Helix approach.

Session-Aware Radix Cache

Targeting agentic and RL rollout workloads, SGLang introduces a session-reference-aware unified Radix cache. Requests can carry a stable session_id, enabling cache eviction to recognize prefixes still referenced by active sessions rather than evicting purely by cache policy, with references released via /close_session. This is particularly valuable for long-running agent scenarios.

Summary: A Comprehensive Evolution of a Production-Grade Inference Engine

SGLang v0.5.17 is an update with both breadth and depth. On the model support front, it keeps pace with the most cutting-edge models like Kimi K3 and MiniMax-H3, cementing its reputation for day-0 serving capability. On the engineering side, the introduction of a Rust frontend, 5–7x loading speedups, the DWDP parallelism strategy, and consistent optimizations across NVIDIA and AMD all point toward a more efficient, production-ready inference engine.

For teams building large model serving infrastructure, this release delivers tangible gains across three production-critical dimensions: cost (memory savings), availability (fast recovery), and throughput (DWDP, MegaMoE) — making it well worth a serious evaluation for upgrading.