Buoyant Open-Sources Kinference: An Inference Control Plane for vLLM on K8s

Buoyant has released Kinference—an open-source inference control plane for vLLM on Kubernetes, licensed under Apache 2.0. It brings the traffic governance ideas accumulated during the service mesh era into the LLM inference space, attempting to solve an increasingly practical problem: once inference clusters scale up, simply throwing more GPUs at the problem and tuning vLLM parameters is no longer enough.

Inference Serving Is Becoming a "Governance" Business

Over the past year, vLLM has become almost the de facto standard for self-hosted inference serving, but it is fundamentally an inference engine, not a cluster-level scheduling system. When multiple teams, multiple models, and multiple tenants share a single Kubernetes cluster, who decides which instance a request should land on? Who handles fault isolation and capacity allocation? The answer to these questions is usually "write your own layer."

Kinference positions itself as exactly this missing middleware layer. According to the official description, its core capabilities center on three areas: declarative routing policies, KV-cache-aware load balancing, and per-tenant metrics. These three map precisely to the most error-prone aspects of inference cluster operations.

Declarative Routing: Taking Policies Out of the Code

In the traditional approach, request distribution logic is often hardcoded into gateways or application code, so changing a policy requires a release. Kinference adopts declarative configuration, meaning routing rules can be version-controlled, audited, and reused just like any other Kubernetes resource. For clusters hosting multiple models, this can significantly reduce the operational risk that comes with policy changes.

KV-Cache Awareness: Load Balancing Can't Just Look at Connection Counts

This is Kinference's most technically substantive feature. The performance bottleneck in LLM inference is often not computation but KV cache hit rate—if a request is randomly routed to an instance without the relevant context, the prefill phase has to recompute everything, and latency immediately spikes. Kinference's load balancing takes KV cache state into account, directing requests to instances more likely to have a cache hit.

In Buoyant's own benchmark, the figure cited is a 95% reduction in median TTFT (time to first token). This number should be viewed with caution, since it comes from the vendor's own test environment, and the specific workload model and comparison baseline were not fully disclosed. But the direction aligns with industry consensus: cache-aware routing improves inference latency at an order-of-magnitude level, not as a marginal optimization.

The Relationship Between KV Cache and Prefill: LLM inference is typically divided into two phases—the prefill phase processes the full input at once and generates the KV cache, while the decode phase generates output token by token. The KV cache stores the attention key-value pairs of historical tokens to avoid redundant computation. In multi-turn conversations or long-context scenarios, if a new request can reuse existing cache, the prefill computation can be dramatically reduced; otherwise it must be computed from scratch, and TTFT rises significantly. The core of KV-cache-aware routing is to read cache prefix information from each instance at the scheduling layer and preferentially route requests to instances with high cache overlap.

Per-Tenant Metrics: A Prerequisite for Multi-Team Sharing

In a shared cluster, without observability data broken down by tenant, capacity disputes are almost unavoidable. Kinference provides per-tenant metrics, allowing platform teams to see clearly how much inference resource each tenant consumes and how its latency performs. This is a necessary condition for a multi-tenant inference platform to move from "functional" to "operable."

How It Differs from Existing Solutions

The Kubernetes ecosystem already has projects like Gateway API Inference Extension doing similar things, and service mesh vendors are increasingly extending their traffic governance capabilities into the inference space. Kinference's differentiation lies in its explicit binding to vLLM rather than being a generic inference abstraction layer. The benefit of this trade-off is deeper access to vLLM's internal state (such as the KV cache); the cost is ecosystem lock-in.

For environments already running vLLM in production, this binding is actually an advantage; for teams still in the selection phase, flexibility needs to be weighed.

Gateway API Inference Extension: This is a project advanced by the Kubernetes SIG Network community, aimed at defining a standardized gateway extension model for inference workloads—for example, expressing model routing and instance selection through resources like InferencePool and InferenceModel. It takes a generic, vendor-neutral approach and does not bind to any specific inference engine. Kinference's relationship with it is more complementary than substitutive: the former can be seen as a vertical implementation of similar problems in the vLLM context, trading generality for direct access to the engine's internal state (such as the KV cache).

Worth Watching but Needs Validation

An open-source release is only the starting point. The value of an inference control plane depends heavily on compatibility with specific inference engine versions and on stability under high concurrency. Whether the 95% TTFT improvement can be reproduced under real-world workloads and how much overhead the multi-tenant metrics incur are questions the community will need to answer through actual deployments. The Apache 2.0 license lowers the barrier to trial; what remains to be seen is the pace of adoption.

Comparability of TTFT Benchmarks: TTFT (Time To First Token) is affected by multiple factors including input length, concurrency, model size, GPU model, and batching strategy, and can vary several-fold across different test configurations. If the 95% reduction comes from comparing two extreme scenarios—"complete cache miss" versus "complete cache hit"—it is closer to a theoretical ceiling than to typical business gains. When evaluating such figures, one should pay attention to whether the comparison baseline is random routing within the same cluster, whether the workload includes a realistic multi-turn conversation distribution, and whether complete P50 and P99 percentiles are reported.

One-line Summary

Buoyant open-sources Kinference, an inference control plane for vLLM on Kubernetes.

Paragraph Summary

Buoyant has released Kinference, an Apache 2.0-licensed open-source project positioned as an inference control plane for vLLM on Kubernetes, filling the middleware gap between inference engines and cluster scheduling. Its core capabilities include declarative routing policies, KV-cache-aware load balancing, and per-tenant metrics, addressing the three pain points of policy management, latency optimization, and multi-tenant operability respectively. The official benchmark claims a 95% reduction in median TTFT, but test details were not fully disclosed and require validation through actual deployment. Kinference explicitly binds to vLLM rather than building a generic abstraction layer, enabling deeper use of the engine's internal state at the cost of ecosystem lock-in; this is an advantage for teams already running vLLM in production, while teams still evaluating options need to weigh flexibility.