llm-d 1.0 Released: Cache-Aware Routing Unifies TPU and GPU Inference

Large language model inference is moving from single-machine experiments to clustered production deployments, and the complexity of scheduling and routing is often the key to architectural success or failure. llm-d 1.0, jointly released by Google, Red Hat, and IBM, targets exactly this pain point: a Kubernetes-native open-source distributed inference stack that runs across vLLM, SGLang, and heterogeneous TPU/GPU hardware, introducing cache-aware routing and multi-tenant flow control.

Cache-Aware Routing: Making Prefix Reuse Actually Work

The most technically significant capability of llm-d 1.0 is cache-aware routing. In LLM inference, the KV Cache hit rate directly determines first-token latency and overall throughput, yet traditional load balancers typically round-robin by request count or connection count and cannot perceive which backend node has already cached a similar prefix. llm-d incorporates cache state into routing decisions, directing requests to backend instances that already hold the relevant context, thereby reducing redundant computation.

This approach yields clear benefits for multi-turn conversations, RAG (retrieval-augmented generation), and batch Agent scenarios that share system prompts. It complements vLLM's PagedAttention and SGLang's RadixAttention mechanistically—the latter two address single-node memory utilization, while llm-d addresses cross-node request flow.

KV Cache and PagedAttention / RadixAttention

KV Cache (key-value cache) is a core optimization in Transformer autoregressive inference: each time a token is generated, the model needs the Key and Value vectors of all previous tokens. Without caching, every step would require full recomputation at extremely high cost. Storing computed K/V in GPU memory reduces the computational complexity of the generation phase from quadratic in sequence length to linear. However, KV Cache size grows linearly with sequence length and concurrency, quickly making GPU memory the bottleneck.

vLLM's PagedAttention borrows the idea of virtual memory paging from operating systems, dividing the KV Cache into fixed-size blocks allocated on demand, eliminating memory fragmentation and allowing a single GPU to host more concurrent sequences. SGLang's RadixAttention uses a radix tree to organize prefixes shared across multiple requests, automatically reusing cache blocks corresponding to identical system prompts or conversation histories. Both focus on memory utilization efficiency within a single node, whereas llm-d's cache-aware routing extends this perspective to the cluster: it needs to know which prefixes each backend instance holds cached in order to direct new requests to the node with the highest hit rate. This requires sharing cache metadata between the scheduling layer and the inference engine, which is the essential difference from ordinary load balancers.

Agent Awareness and Multi-Tenant Flow Control: Closing Production Gaps

Beyond routing, version 1.0 adds several engineering capabilities for production environments. Multi-tenant flow control lets different teams or applications sharing the same inference cluster obtain isolated quotas and priorities, preventing a single high-concurrency task from overwhelming the whole system. Zero-downtime upgrades address a long-standing operational pain point: inference services typically carry session state, and rolling restarts can easily cause request loss. llm-d handles canary rollout and draining at the scheduling layer, reducing the risk of version iterations.

Agent-aware routing is another notable design. Multi-step Agent workflows exhibit obvious call-chain dependencies and context drift, and call-graph-aware routing can reduce cross-node state synchronization overhead. This shows that the project is positioned not merely as an "inference service mesh" but aims to cover the orchestration needs of the Agent era.

Quota and Priority Mechanisms in Multi-Tenant Flow Control

Multi-tenant flow control in inference clusters is typically implemented through hierarchical token buckets or weighted fair queuing: each tenant (team, application, or API key) is allocated request rate and concurrency limits, and scheduling follows priorities when resources are tight. Unlike rate limiting for ordinary web services, LLM inference is measured not only in request count but also in input tokens, output tokens, and estimated KV Cache footprint, so the quota model requires multi-dimensional accounting. The goal of isolation is to prevent a single high-concurrency batch task from saturating GPU memory and causing first-token latency spikes for other tenants, while ensuring high-priority online sessions still obtain resources under overload. llm-d pushes this capability down to the scheduling layer, meaning tenant policies can take effect uniformly across different engines such as vLLM and SGLang without repeated configuration on each inference instance.

One Release, Three Endorsements: Ecosystem Signals Stronger Than Individual Features

The collaboration among Google, Red Hat, and IBM is itself an important signal. Red Hat represents the Kubernetes ecosystem and enterprise-grade delivery capability, IBM brings hybrid cloud customer scenarios, and Google Cloud commands TPU hardware and the GKE platform. Joint backing from all three means llm-d has the opportunity to become the de facto standard for cross-cloud, cross-chip inference scheduling layers, rather than a proprietary optimization tool for any single vendor.

For teams using TPUs, the unified abstraction llm-d provides is especially valuable: using the same set of scheduling policies across mixed GPU and TPU clusters can reduce the cost of maintaining two separate inference infrastructures for different accelerators. However, performance tuning under heterogeneous hardware still requires empirical validation—a unified API does not equal unified performance.

The Difference Between Kubernetes-Native and an Inference Service Mesh

"Kubernetes-native" means llm-d extends the K8s API in the form of custom resource definitions (CRDs) and controllers, reusing its declarative configuration, service discovery, health checks, and autoscaling mechanisms rather than building a separate control plane outside K8s. This differs from the early practice of simply wrapping inference services as Deployments: llm-d needs to be aware of model replicas, cache distribution, and request routing state above the Pod level, which introduces semantics similar to the Gateway API Inference Extension. Understanding it as an "inference service mesh" helps grasp its positioning—it handles east-west traffic governance between services (routing, retries, circuit breaking, tenant isolation) rather than north-south ingress gateways. This also explains why Red Hat's role in the three-party collaboration is critical: enterprise-grade K8s distributions and Operator delivery experience are essential to bringing the scheduling layer into production environments.

Adoption Advice: Who It's For and What to Watch Out For

Teams that already have a Kubernetes foundation and whose inference workloads have reached the multi-node scale are the most direct target users of llm-d. If you are still at the single-GPU or single-node stage, the marginal benefit of introducing a scheduling layer is limited and instead adds operational complexity. When evaluating, focus on three points: the actual improvement in cache hit rate, the behavior of multi-tenant isolation under pressure, and the compatibility of zero-downtime upgrades with long-lived streaming outputs.

Competition in open-source inference stacks is shifting from "single-engine performance" to "cluster-level scheduling and resource efficiency." llm-d 1.0 brings caching, tenancy, upgrades, and Agent workflows into a single abstraction, aligning with production needs in direction; whether it truly becomes a standard depends on the pace of community adoption and consistent performance across hardware backends.

One-line Summary

Google, Red Hat, and IBM jointly released the Kubernetes-native distributed inference stack llm-d 1.0, filling in production-grade scheduling capabilities with cache-aware routing at its core.

Paragraph Summary

llm-d 1.0 is an open-source distributed LLM inference stack jointly released by Google, Red Hat, and IBM, positioned as a Kubernetes-native cluster-level scheduling and routing layer that runs across vLLM, SGLang, and heterogeneous GPU/TPU hardware. Its core technology is cache-aware routing: incorporating the KV Cache state of each backend instance into routing decisions and directing requests to nodes that already hold similar prefixes, thereby reducing first-token latency and improving throughput, with significant benefits for multi-turn conversations, RAG, and Agent scenarios with shared system prompts. Version 1.0 also adds production capabilities including multi-tenant flow control, zero-downtime upgrades, and Agent-aware routing. The three-party endorsement is seen as an important signal of its potential to become the de facto standard for cross-cloud, cross-chip inference scheduling, but actual performance under heterogeneous hardware still requires empirical validation.