Production-Grade LLM Inference Deployment: A Practical K8s Handbook

Why Build Your Own LLM Inference Infrastructure

As large language models (LLMs) become increasingly embedded in enterprise applications, more teams are considering migrating their inference services from third-party APIs to self-hosted infrastructure. An engineer recently shared their complete hands-on experience building an LLM inference platform from scratch at their organization (original post at gd03.me/writings/inference-infra), offering a valuable "production runbook" for teams facing similar challenges.

LLM inference refers to the process of deploying a trained model as a service that receives user inputs and generates outputs. Unlike the training phase, the core challenge during inference is maximizing throughput while meeting latency requirements. The autoregressive nature of LLM generation (producing tokens one at a time) introduces inherent sequential dependencies, fundamentally different from traditional inference tasks like image classification that complete in a single forward pass. This distinction is the root cause behind the emergence of various specialized inference optimization techniques.

The core motivations for building self-hosted inference infrastructure typically come down to three factors: data privacy and compliance, long-term cost control, and service controllability. When your workload involves sensitive data, call volumes reach a certain scale, or you need fine-grained control over latency and availability, deploying inference services on your own Kubernetes cluster becomes a rational choice.

Key Considerations for Deploying Inference Services on Kubernetes

GPU Resource Scheduling and Management

The primary challenge of running LLM inference on Kubernetes is efficient GPU resource scheduling. Unlike traditional CPU workloads, GPUs are scarce and expensive resources that need to be exposed to the cluster through mechanisms like the NVIDIA Device Plugin, combined with node affinity, taints & tolerations to ensure inference Pods are correctly scheduled onto GPU nodes.

The NVIDIA Device Plugin is a DaemonSet that runs on every GPU node, responsible for reporting the number and status of available GPUs to Kubernetes' kubelet. It implements the Kubernetes Device Plugin framework interface, allowing GPUs to be requested by Pods through resource requests (nvidia.com/gpu: 1) just like CPU and memory. Under the hood, it relies on the NVIDIA Container Toolkit (formerly nvidia-docker) to ensure containers can properly access GPU drivers and the CUDA runtime.

For multi-GPU scenarios, you also need to consider deployment topology for tensor parallelism and pipeline parallelism, avoiding performance degradation from cross-node communication. Tensor parallelism splits individual model layers (such as attention heads or FFN layers) across multiple GPUs for parallel computation, requiring extremely high inter-GPU communication bandwidth (typically NVLink or NVSwitch), making it best suited for multi-GPU setups within a single node. Pipeline parallelism assigns different layers to different GPUs, with data flowing sequentially through each card, requiring relatively lower communication bandwidth and suitable for cross-node deployment. In production, models with 70B+ parameters typically require 4-8 GPUs with tensor parallelism to meet the memory and latency requirements for inference. Proper configuration of resource requests and limits is the foundation for ensuring service stability.

Inference Engine Selection

In production environments, running raw model forward passes can no longer meet throughput demands. Mainstream approaches introduce specialized inference engines such as vLLM, TGI (Text Generation Inference), or TensorRT-LLM. These engines dramatically improve GPU utilization and overall throughput through techniques like PagedAttention, continuous batching, and KV Cache optimization.

PagedAttention is the core innovation proposed by the vLLM project, inspired by the virtual memory paging mechanism in operating systems. Traditional inference frameworks pre-allocate contiguous GPU memory for each request's KV Cache. Since sequence lengths are unpredictable, memory is often reserved at maximum length, causing significant memory fragmentation and waste (research shows waste rates of 60-80%). PagedAttention divides KV Cache into fixed-size blocks managed through a block table mapping, allowing non-contiguous physical memory to store logically contiguous KV Cache. This dramatically improves memory utilization, enabling a single GPU to serve more concurrent requests.

Continuous batching (also called iteration-level batching) solves the efficiency problem of traditional static batching. Static batching requires waiting for an entire batch of requests to finish generating before accepting new ones, leaving the GPU idle after short sequences complete while waiting for longer ones. Continuous batching checks at the end of each generation step whether any requests have completed, and if so, immediately inserts new requests into the batch, keeping the GPU at high utilization at all times. In practice, this can improve throughput by 2-5x.

In autoregressive generation, KV Cache avoids redundant computation by caching previously computed Key/Value pairs, but its memory consumption scales proportionally with sequence length and concurrency. For example, a single Llama-2 70B request at 4096 context length requires approximately 2.5GB of GPU memory for KV Cache. Optimizing KV Cache (through techniques like GQA grouped-query attention, quantized KV Cache, prefix sharing, etc.) directly determines the maximum concurrency a single card can serve.

Engine selection requires balancing throughput, latency, ease of use, and hardware compatibility. For example, vLLM has become the go-to choice for many teams due to its excellent throughput and active community, while TensorRT-LLM offers advantages in extreme performance scenarios but comes with higher deployment complexity.

Engineering Practices: From Prototype to Production

Elastic Scaling and Load Balancing

Inference traffic often exhibits clear peaks and valleys. A fixed replica deployment simply wastes resources during off-peak hours or gets overwhelmed during peaks. Kubernetes HPA (Horizontal Pod Autoscaler) can automatically scale based on custom metrics such as GPU utilization and request queue length.

Kubernetes' native HPA only supports CPU and memory metrics. To scale based on GPU utilization or request queue length, you need to expose custom metrics through the Custom Metrics API or External Metrics API. A typical implementation path is: inference service exposes Prometheus metrics endpoint → Prometheus scrapes and stores metrics → Prometheus Adapter converts metrics to Kubernetes custom metrics API → HPA controller makes scaling decisions accordingly. KEDA (Kubernetes Event-driven Autoscaling) is another popular choice, supporting richer trigger sources and more flexible scaling strategies.

However, LLM inference services have long cold start times (model loading can take tens of seconds or even minutes), so scaling strategies need to build in buffers to avoid request backlogs from delayed scale-ups. Some teams use pre-warmed replicas or staged scaling approaches to mitigate cold start issues.

Observability and Monitoring

Production-grade services cannot function without a comprehensive observability stack. Beyond standard CPU/memory monitoring, inference services need to focus on key metrics including GPU utilization, memory usage, Time To First Token (TTFT), Time Per Output Token (TPOT), and throughput (tokens/s).

Time To First Token (TTFT) measures the time from when a request is sent to when the first generated token is received, directly impacting the user's perception of response speed — especially critical in streaming chat scenarios. Time Per Output Token (TPOT) measures the generation interval for each subsequent token, determining the fluidity of text output. In production, TTFT is typically expected to be within 500ms-2s, while TPOT needs to be kept at 30-80ms for a smooth typewriter effect. These two metrics are influenced by different computational characteristics of the prefill and decode phases — the prefill phase is compute-intensive (processing all input tokens in parallel) and determines TTFT; the decode phase is memory-bandwidth-intensive (generating tokens one by one) and determines TPOT. Each requires separate optimization.

Building monitoring dashboards with Prometheus + Grafana, combined with alerting rules, enables timely detection of anomalies like out-of-memory (OOM) errors and request timeouts. Log aggregation and distributed tracing help pinpoint root causes of complex issues.

Cost Optimization

Cost is one of the biggest concerns with self-hosted inference. Beyond improving per-card throughput with efficient inference engines, you can further reduce unit inference costs through quantization (e.g., INT8, FP8), model distillation, and hybrid deployment (large models for complex requests, small models for simple requests).

Model quantization converts weights and/or activations from high precision (FP32/FP16) to low precision (INT8/INT4/FP8) representation, dramatically reducing memory usage and compute requirements with acceptable accuracy loss. GPTQ and AWQ are currently the mainstream weight-only quantization methods, typically halving model size (FP16→INT8) or reducing it to a quarter (FP16→INT4) with nearly no loss. FP8 quantization is a precision format natively supported by next-generation GPUs like the H100, offering better dynamic range than INT8 with less accuracy degradation. Quantization not only reduces memory requirements (allowing larger models to run on fewer GPUs) but also improves compute throughput (e.g., INT8 operations on A100 achieve 2x the throughput of FP16).

Leveraging Spot instances or preemptible GPU nodes for batch processing tasks that can tolerate interruptions is another common cost-reduction strategy.

Key Takeaways

The value of this runbook lies in the fact that it's not theoretical — it comes from the author's first-hand experience navigating pitfalls and iterating in a real production environment. For teams planning to build their own LLM inference platform, several lessons are worth noting:

• Don't optimize prematurely: Start by getting the full pipeline running with a mature inference engine (like vLLM), then optimize based on actual bottlenecks.
• Prioritize observability: GPU-related metrics must be included in monitoring — this is a prerequisite for maintaining service quality.
• Embrace the cloud-native ecosystem: Kubernetes' scheduling, scaling, and self-healing capabilities significantly reduce operational burden, but require customized configurations for GPU and large model characteristics.

Overall, deploying LLM inference on Kubernetes has moved from the "can we do it" phase to the "how do we do it well" phase. As open-source tools like vLLM and KServe mature, the barrier to building self-hosted inference infrastructure is rapidly lowering, giving more enterprises complete control over their AI capability stack. KServe (formerly KFServing), as a standardized model serving framework on Kubernetes, provides complete abstractions from model storage, inference runtimes, and autoscaling to traffic routing. It supports serverless mode (scaling to zero to save resources) as well as advanced deployment strategies like canary releases and A/B testing. For LLM scenarios, KServe integrates inference engines like vLLM through custom runtimes, enabling teams to manage different types of model services on a unified platform, further reducing the difficulty of going from experimentation to production.