vLLM Deployment of Kimi K3 Achieves 2.2–2.8x Throughput Gain: A Performance Optimization Breakdown

A Significant Leap in Inference Performance

The deployment of Kimi K3 on vLLM has received a notable performance upgrade. According to official benchmark results on B300 hardware, the new optimizations deliver a 2.2 to 2.8x improvement in throughput compared to v0.27.1. For large-scale model inference services, gains of this magnitude translate directly into lower per-inference costs and higher concurrent request capacity.

This achievement is not the result of a single breakthrough, but rather a systematic engineering effort targeting three critical areas: scheduling, KDA state handling, and MoE kernels. The team also acknowledged contributions from the vLLM community in advancing Kimi K3's performance, underscoring that this is the product of open-source collaboration.

Three Core Optimization Directions

Scheduling Layer: More Efficient Request Orchestration

Scheduling is the first bottleneck in inference service performance. Under high-concurrency scenarios, how requests are batched, and how latency and throughput are balanced, directly determines hardware utilization. vLLM has long been known for its continuous batching mechanism, and the scheduling optimizations for Kimi K3 push this further — squeezing more scheduling efficiency out of high-end accelerators like the B300 by reducing idle wait times and resource fragmentation.

Continuous Batching is vLLM's core scheduling mechanism, distinct from traditional Static Batching. Static batching requires an entire batch to complete before new requests can be admitted, wasting GPU compute while waiting; continuous batching allows new requests to be dynamically inserted after each decoding step, keeping GPU utilization near saturation. However, under extremely high concurrency or when request sequence lengths vary significantly, the scheduler still faces challenges such as long-tail requests blocking shorter ones and KV Cache memory fragmentation. Scheduling optimizations targeting B300 hardware likely involve fine-tuning batch packing strategies and preemption logic to match the hardware's higher memory bandwidth and compute density characteristics.

KDA State Handling: Reducing State Management Overhead

KDA (Kimi's attention/state mechanism) state handling is another focus of this optimization. In long-sequence and multi-turn conversation scenarios, reading, writing, and maintaining state can become a hidden bottleneck. By restructuring the state handling path to eliminate unnecessary memory copies and synchronization overhead, overall throughput improves while correctness is preserved. This kind of low-level optimization is rarely visible to end users, yet it is an indispensable part of the performance improvement curve.

MoE Kernels: Accelerating Mixture-of-Experts Computation

Kimi K3 uses a Mixture-of-Experts (MoE) architecture, meaning only a subset of expert networks is activated during each forward pass. MoE routing and expert computation are highly sensitive to kernel implementation — expert dispatch, load balancing, and efficient operator scheduling on the GPU all significantly affect final performance. Dedicated optimizations to the MoE kernels are one of the key drivers behind the order-of-magnitude throughput gains in this release.

The core of the Mixture-of-Experts (MoE) architecture is its routing mechanism: for each input token, a lightweight router network determines which "experts" to activate (typically a Top-K selection), while all other experts' parameters are skipped entirely during that forward pass. This decouples parameter count from actual compute — the total model size can be very large, while the floating-point operations (FLOPs) per inference remain relatively low. However, the engineering challenge in MoE lies in the dispatch-gather operation: tokens routed to different experts require an irregular memory access pattern on the GPU that is extremely sensitive to kernel implementation. Load imbalance across experts (some overloaded, some idle) also drags down overall utilization. An efficient MoE kernel must therefore address both computation regularity and load balancing simultaneously.

Reproducible Benchmarks

Noteably, the team didn't stop at presenting impressive numbers — they also provided complete benchmarking methodology and reproduction commands. This level of transparency is not always standard practice in the industry, and it allows third-party developers and teams to verify results in their own environments rather than relying solely on vendor claims.

That said, a note of caution is warranted: the 2.2–2.8x gains were achieved on specific B300 hardware under specific benchmark workloads. Actual benefits will vary depending on model configuration, request distribution, sequence length, and hardware type. For teams planning to deploy Kimi K3, running the official reproduction steps against your own workloads will yield the most production-relevant assessment.

The B300 is NVIDIA's Blackwell-architecture server-grade accelerator. Compared to the previous generation H100/H800, it offers significantly higher memory capacity, memory bandwidth, and FP8 compute throughput, making it particularly well-suited for large-scale MoE model inference. Benchmarks on this class of hardware typically use Output Tokens per Second or Requests per Second as throughput metrics, supplemented by Time to First Token (TTFT) and Time per Output Token (TPOT) to assess response latency. Because the B300's hardware characteristics differ from the H100, kernel and scheduling optimizations tuned for B300 may not yield the same gains on other hardware — which is precisely why the official results are specifically labeled as "B300 benchmarks."

What This Means for Developers and Deployment Teams

For teams that rely on large model inference services, the value of these optimizations is straightforward. Doubling throughput means serving more users within the same hardware budget, or reducing hardware spend for the same workload. At a time when inference cost remains a core challenge in large model commercialization, this collaboration between vLLM and Kimi K3 provides a concrete example of cost reduction in practice.

From an ecosystem perspective, this also reaffirms vLLM's position as the leading open-source inference framework. The pattern of performing deep, model-specific optimizations (as with Kimi K3) and contributing results back to the community is becoming the standard mode of iteration for large model infrastructure. For engineering teams evaluating inference frameworks, the active community and sustained model adaptation capability behind a framework are often more important long-term considerations than peak performance at any given moment.

Summary

This performance upgrade for Kimi K3 on vLLM is the result of coordinated optimizations across scheduling, state handling, and MoE kernels, delivering 2.2–2.8x throughput improvements on B300 benchmarks with fully reproducible test methodology. For developers focused on inference efficiency and deployment costs, this is a case study well worth diving into via the official technical breakdown. Interested readers can refer to the official in-depth analysis for complete details.