SGLang
SGLang (Structured Generation Language) is an open-source inference framework for large language models, designed to improve both execution efficiency and programming flexibility in LLM serving. Key features include efficient KV cache reuse via RadixAttention, support for structured output constraints such as JSON mode, and a concise front-end programming interface for constructing complex multi-step inference pipelines. SGLang is compatible with multiple mainstream model architectures and GPU hardware, suitable for high-performance inference deployment in both academic research and production environments.
Core Facts
SGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.
SGLang is a high-performance framework designed for large language model inference, developed by researchers from UC Berkeley and other institutions.
SGLang (Structured Generation Language) is a high-performance serving framework for large language models and vision-language models.
Timeline (last 90 days)
SGLang added the /v1/decisions endpoint in its nightly version, and vLLM merged the /v1/systemone endpoint.
In the SGLang and vLLM inference engines, tokenization is performed by high-speed libraries written in Rust or C++, but each call must go through the Python binding layer.
In the SGLang logs, each health check appears as a one-token prefill on the GPU, but the request counter does not move, indicating that health checks consume compute without producing user value and without appearing in request metrics.
If the bill is split at the moment the health check passes (386.7 seconds), startup appears to account for 60%; but the first user actually gets an answer at about 563 seconds, by which point 88% of the bill has already been consumed.
The 237.7-second prefill CUDA graph capture is the largest single block of overhead in the entire SGLang cold-start timeline.
The measured timeline was: billing starts at 0 seconds, prefill CUDA graph capture completes at about 371 seconds (taking 287.7 seconds), /health returns 200 at 386.7 seconds, the first request completes at about 563 seconds (taking about 176 seconds), the second request completes at about 579 seconds, and billing stops at 639.6 seconds.
Running SGLang 0.5.21 on a single T4 in Colab to load the Qwen2.5-0.5B-Instruct fp16 model, in one complete cold-start measurement, 88% of the billed time was spent before the first user request completed.
At startup, vLLM typically performs CUDA graph capture only for the decode phase, and has a more aggressive trimming strategy for the set of captured batch sizes, so its cold-start graph capture phase is often shorter than SGLang's.
Eight days later, Cloudflare cut Clef-flash from $0.09 to $0.038 (already below Jev), migrated Clef's inference to SGLang for a 1.7 to 2.0 times median speedup, and released clef-omni; Clef itself still remains at $0.24.
Mainstream open-source inference engines such as vLLM and SGLang have implemented cross-request KV reuse, some support finer-grained prefix tree matching, and they come with cache eviction strategies such as LRU and TTL.
40 more timeline events
All Facts (20)
SGLang's core innovation is the RadixAttention technology, which achieves automatic KV Cache reuse through a prefix tree structure.
85%VerifiedSGLang is a high-performance framework designed for large language model inference, developed by researchers from UC Berkeley and other institutions.
80%VerifiedSGLang (Structured Generation Language) is a high-performance serving framework for large language models and vision-language models.
75%VerifiedSGLang achieves efficient KV Cache reuse through innovations such as RadixAttention, and can deliver several-fold throughput improvements when handling complex multi-turn dialogue and Agent Loop scenarios.
75%VerifiedIn an RTX 5070 (12GB VRAM) configuration, the local Qwen3 9B model's output speed reached approximately 70–76 tokens/second.
70%UnverifiedSGLang由斯坦福团队开发，专注于结构化输出和复杂推理链的高效执行
90%UnverifiedSGLang由UC Berkeley LMSYS团队开发，于2024年初开源
85%UnverifiedDeepGEMM是DeepSeek专为其MoE模型设计的高性能矩阵乘法库，支持bf16和fp8等低精度格式
85%UnverifiedNIXL是SGLang用于Prefill-Decode分离部署的关键通信组件，负责在Prefill节点和Decode节点之间传输KV Cache状态
85%UnverifiedSGLang由UC Berkeley团队主导开发，是目前与vLLM并列的两大主流开源LLM推理框架之一
85%UnverifiedSGLang通过优化的张量传输协议和零拷贝技术降低KV Cache跨节点迁移延迟
85%UnverifiedSGLang的RadixAttention对于共享系统提示（system prompt）的场景可带来数倍吞吐量提升
75%UnverifiedBy copying CPU weight views to contiguous memory before H2D transfer, full model loading time was reduced from approximately 35 minutes to 6 minutes 20 seconds, achieving a 5.6× speedup.
70%UnverifiedThe SGLang project is maintained by the sgl-project team.
70%UnverifiedSGLang's RadixAttention technology can automatically reuse KV Cache across multiple requests, managing shared prefixes through a radix tree structure to reduce redundant computation.
70%UnverifiedSGLang 引入了 RadixAttention 技术，在某些基准测试中推理速度可比 vLLM 等主流推理框架提升 2-5 倍
70%UnverifiedSGLang 的开源属性使其能够比 NVIDIA 主导的 TensorRT-LLM 更快速地集成 AMD ROCm 生态的新特性
70%UnverifiedSGLang对MoE模型的专家并行支持比vLLM更为成熟，使其成为部署Mixtral、DeepSeek、Kimi K2.6等MoE架构模型的首选框架之一
70%UnverifiedDocker部署vLLM/SGLang过程中会产生大量缓存，几个小时就能写入100GB以上
60%UnverifiedIn the SGLang and vLLM inference engines, tokenization is performed by high-speed libraries written in Rust or C++, but each call must go through the Python binding layer.
50%