#SGLang

40 related articles

Jul 12, 2026·2 min

SGLang Integrates DSpark: How Confidence-Driven Speculative Decoding Cracks the High-Concurrency Failure Problem

SGLang officially integrates DSpark, solving the core pain point of speculative decoding failure under high-concurrency batches via confidence-driven variable-length verification. Supports Qwen3 and DeepSeek-V4, hitting 383.7 tok/s on B300.

Jul 12, 2026·4 min

LingBot-World 2.0 Meets SGLang: A Deep Dive into the Evolution of Real-Time Interactive World Models

SGLang-Diffusion now officially supports LingBot-World 2.0, delivering leaps in resolution and temporal consistency. With live sessions, chunked streaming, and camera control, world models achieve low-latency controllable interaction.

Jul 11, 2026·3 min

TB/s Memory Tier Accelerates SGLang: How X-Mem Reduces TTFT by 6.7x

Netpreme integrates X-Mem™ MPU into SGLang HiCache, achieving up to 6.7× TTFT reduction and 33–50% TPS gains at 98% prefix cache hit rates. Here's the technical breakdown.

Jul 11, 2026·2 min

AI Agents for SGLang Kernel Optimization: Real-World Performance Gains and Engineering Methodology

SGLang's team converted expert knowledge into agent skills, achieving 71.4% throughput gains, TTFT reduced from 456ms to 168ms. A deep dive into agent-assisted kernel optimization methodology.

Tutorial: Deploying a PD-Disaggregated SGLang Multi-Node Inference Cluster on AMD GPUs

Tutorials

May 30, 2026·2 min

Tutorial: Deploying a PD-Disaggregated SGLang Multi-Node Inference Cluster on AMD GPUs

Learn how to deploy a PD-disaggregated SGLang inference cluster on AMD GPUs using a single config file, boosting LLM throughput and latency performance.

SGLang v0.5.12.post1 Released: DeepSeek V4 Stability Fixes and Blackwell Adaptation

Tech Frontiers

May 30, 2026·2 min

SGLang v0.5.12.post1 Released: DeepSeek V4 Stability Fixes and Blackwell Adaptation

SGLang v0.5.12.post1 stability patch details: 12 critical fixes covering DeepSeek V4 garbled text and crashes, NIXL PD disaggregated inference logic, Blackwell B300 adaptation, and cold start optimization.

SGLang Enters Finance: How AI Inference Infrastructure Is Reshaping Wall Street

Industry Insights

May 30, 2026·2 min

SGLang Enters Finance: How AI Inference Infrastructure Is Reshaping Wall Street

SGLang co-hosts a finance AI inference event with Crusoe AI and Cloudflare, exploring LLM inference deployment in trading, risk management, and compliance — signaling Wall Street's shift to production-grade AI infrastructure.

Cloudflare Contributes Critical KV Cache and Mooncake Fixes to SGLang

Tech Frontiers

May 30, 2026·1 min

Cloudflare Contributes Critical KV Cache and Mooncake Fixes to SGLang

Cloudflare contributes decode KV cache offload and Mooncake recovery fixes to SGLang, resolving garbled output under high concurrency for Kimi K2.6 and enabling automatic fault recovery in distributed inference.

SGLang Hosts Agent Loops Office Hour, Focusing on Agentic Loop Architecture Optimization

Tech Frontiers

May 30, 2026·1 min

SGLang Hosts Agent Loops Office Hour, Focusing on Agentic Loop Architecture Optimization

SGLang team hosts an Agent Loops Office Hour exploring inference optimization for agentic loops, covering KV Cache reuse, low-latency multi-turn dialogue, and tool calling techniques.

Jul 28, 2026·3 min

A 9B Model Beats Frontier LLMs After Just $500 in RL Fine-Tuning

Just $500 in RL fine-tuning enables a 9B open-source model to outperform frontier LLMs on catalog review tasks. Analysis of when small-model RL works and its enterprise implications.

Jul 25, 2026·3 min

D-Flash: The Secret Behind 3.5x LLM Inference Speedup with Diffusion Drafting

D-Flash uses fast diffusion parallel drafting and target feature KV injection to solve the autoregressive Drafter latency bottleneck in speculative decoding. 16 tokens in just 6ms, up to 3.5x speedup on HumanEval, surpassing EAGLE3 and MTP.

Jul 25, 2026·2 min

D-Flash: How Diffusion Drafting Delivers 3.5x Faster LLM Inference

D-Flash solves the autoregressive drafter latency bottleneck in speculative decoding via fast diffusion parallel drafting and target feature KV injection. 16 tokens in just 6ms, up to 3.5x speedup on HumanEval, beating EAGLE3 and MTP.

Jul 25, 2026·3 min

Laguna S 2.1: The Open-Source Coding Model That's 14x Smaller Yet Stronger

Poolside releases Laguna S 2.1, an open-source agentic coding model: 118B total params with only 8B active, scoring 40.4% on DeepSWE — ~4.5x DeepSeek V4 Pro Max. Supports 1M-token context, deployable on a single workstation.

Jul 25, 2026·3 min

DeepSeek Open-Sources DeepSpec: A Full-Stack Speculative Decoding Library with 60–85% Inference Speedup

DeepSeek open-sources DeepSpec, a full-stack speculative decoding library delivering 60–85% inference speedup and up to 661% throughput gains in production. MIT license, 9 checkpoints, 9 benchmarks.

Jul 24, 2026·6 min

Hands-On with Panel AI On-Premises Deployment: One-Click Enterprise Adoption and Unified Compute Cluster Scheduling

In-depth review of Panel AI v1.1.1: second-level installation, no-public-IP networking, batch compute cluster management. Learn how enterprise AI on-premises deployment barriers are dramatically lowered.

Jul 16, 2026·3 min

GPUStack in Action: Complete Guide to Day 0 Deployment of DeepSeek-V4-Flash

Deploy DeepSeek-V4-Flash DSpark on 8× H20-141G using GPUStack's SGLang backend on Day 0. Full walkthrough of Web UI config, parameter tuning, and 200 tokens/sec benchmark results.

Jul 14, 2026·2 min

QuantaMind: An Open-Source Tool for Testing AI Agent Reliability Locally

QuantaMind is a free, open-source local AI Agent reliability testing tool using pass^k scoring and deterministic evaluation, supporting Ollama, llama.cpp, vLLM, and more.

Jul 13, 2026·4 min

From Enter to First Token: Decrypting the Full LLM Inference Pipeline (Autoregression, KV Cache & Decoding Strategies)

From pressing Enter to the first character appearing, what happens inside an LLM? This article breaks down autoregressive generation, KV cache acceleration, and decoding strategies like temperature, Top-k, and Top-p.

Jul 12, 2026·3 min

DeepSeek-V4 Flash RL Training Successfully Ported to AMD MI355X GPU: A Complete Breakdown

The Miles team and AMD announce the full port of DeepSeek-V4 Flash RL training to AMD Instinct MI355X GPUs on ROCm, boosting AIME pass@1 from 0.39 to 0.49—a milestone for compute ecosystem diversity.

Jul 11, 2026·6 min

A Deep Dive into the vLLM Inference Framework: Core Principles of Throughput Optimization and Interview Guide

An in-depth analysis of the vLLM inference framework's core principles: from the meaning of throughput (tokens/s), to the bottlenecks of autoregressive generation, to KV Cache, PagedAttention, and continuous batching.