Self-Hosting LLMs on Azure: A Complete Guide to vLLM + AKS + GPU Deployment

Why Self-Host Large Language Models

As large language models (LLMs) become increasingly prevalent in enterprise applications, more teams are facing a critical decision: continue relying on third-party APIs from OpenAI, Anthropic, and others, or deploy models on their own controlled infrastructure? For teams with data compliance requirements, those seeking cost predictability, or those needing deeply customized inference services, self-hosting is becoming an increasingly attractive option.

Recently, a developer (GitHub user shiqs90) shared a complete end-to-end project on Reddit demonstrating how to self-host LLMs on Azure using Kubernetes. The project, available at vllm-serving-aks, covers the full engineering practice from GPU scheduling to cost control, offering high reference value for teams looking to implement private inference services.

Tech Stack Analysis: The vLLM + AKS Combination

The project's core technology choices reflect the mainstream approach to LLM serving today—combining a high-performance inference engine with cloud-native orchestration capabilities.

vLLM: High-Throughput Inference Engine

The project uses vLLM as its inference engine, one of the most popular open-source LLM serving frameworks in the community. vLLM's core advantage lies in its PagedAttention mechanism, which significantly improves GPU memory utilization and dramatically increases inference throughput through continuous batching. Compared to directly using the Transformers library or simple Flask wrappers, vLLM can often deliver several times better concurrent performance in production environments.

PagedAttention Mechanism Deep Dive

PagedAttention is the core innovation published by the vLLM team at the 2023 SOSP conference. In traditional LLM inference, each request's KV Cache (key-value cache, which stores intermediate computation results from Transformer attention layers to avoid redundant calculations) requires pre-allocated contiguous GPU memory blocks. Since the length of generated text is unpredictable, systems often need to reserve memory based on maximum length, resulting in actual utilization typically below 30%. PagedAttention borrows the paging mechanism from operating system virtual memory, splitting the KV Cache into fixed-size blocks that don't need to be contiguous in physical memory, managed through a block table for mapping. This design significantly mitigates memory fragmentation issues—multiple requests can share the same prefix cache blocks (prefix caching), pushing memory utilization close to 100%, thereby supporting higher concurrent request counts on the same hardware.

Continuous Batching vs. Static Batching

Traditional static batching requires all requests in a batch to start and end simultaneously. Since LLM text generation proceeds token by token, shorter responses must wait for the longest response in the batch to finish before resources can be released, causing severe GPU idle waste. Continuous batching (also called iteration-level batching) allows dynamic batch adjustment at each decoding step: when a request finishes generation, a new waiting request is immediately inserted into the batch. This means the GPU runs at maximum capacity at all times. Orca (a scheduling system proposed in 2022) first implemented this concept, and vLLM further optimized scheduling efficiency by combining it with PagedAttention, achieving measured throughput improvements of 14-24x compared to naive inference with HuggingFace Transformers.

AKS: Elastic Scheduling with Managed Kubernetes

The project uses Azure Kubernetes Service (AKS) as the orchestration layer. Choosing Kubernetes over single-machine deployment is primarily for elastic scaling, fault recovery, and standardized operations capabilities. For inference services that need to run long-term and may face traffic fluctuations, the declarative management and horizontal scaling capabilities provided by K8s are indispensable.

Azure Kubernetes Service GPU Node Pool Architecture

AKS manages different types of compute resources through the concept of node pools. In LLM inference scenarios, you typically create at least two node pools: one using standard CPU instances for system components (such as Kubernetes control plane agents, monitoring DaemonSets, etc.), and another using GPU instances (such as NCas_T4_v3, NC_A100_v4, or ND_H100_v5 series) specifically for running inference workloads. The benefit of this separation is that CPU node pools can use low-cost B-series or D-series instances, while GPU node pools can be configured with independent autoscaling policies, scaling down to zero nodes when there are no inference requests, fundamentally eliminating the cost of idle GPUs. AKS also supports integration between the Cluster Autoscaler and Azure's Virtual Machine Scale Sets (VMSS), enabling automatic scaling from zero to multiple nodes, with typical cold start times of approximately 3-8 minutes (depending on GPU instance type availability and image size).

NVIDIA GPU Operator: Standardized GPU Resource Management

Using GPUs on Kubernetes isn't plug-and-play. The project incorporates the NVIDIA GPU Operator, which automates the deployment of GPU drivers, container runtime (container toolkit), device plugins, and monitoring components. This toolchain transforms GPU node initialization from tedious manual configuration into a declarative automated workflow, serving as critical infrastructure for running GPU workloads on K8s.

NVIDIA GPU Operator Component Architecture

The NVIDIA GPU Operator is a Kubernetes Operator that packages the entire software stack required for GPU nodes as a set of coordinated containerized components. Specifically: (1) GPU Driver Container—compiles and loads NVIDIA drivers matching the host kernel within a container, avoiding direct modification of the node OS; (2) NVIDIA Container Toolkit—configures the container runtime (such as containerd) to pass through GPU devices to containers; (3) NVIDIA Device Plugin—registers the nvidia.com/gpu resource type with Kubernetes' resource scheduler, allowing Pods to declare GPU requirements via resources.limits; (4) DCGM Exporter—collects GPU utilization, memory usage, temperature, and other metrics, exposing them in Prometheus format for monitoring and alerting; (5) GPU Feature Discovery—automatically labels nodes with GPU model, driver version, CUDA version, and other attributes, facilitating precise scheduling through nodeSelector. The entire component set is declared and managed through a ClusterPolicy CRD, significantly reducing the complexity of GPU cluster initialization and upgrades.

Key Challenges and Solutions in Production Deployment

Looking at the project's content, the real difficulties lie not in getting a demo running, but in handling the various engineering details during productionization.

GPU Scheduling Configuration

In Kubernetes, GPUs are indivisible scarce resources (by default). How to ensure Pods correctly request GPUs, how to prevent multiple workloads from competing for the same GPU, and how to configure node affinity and taints/tolerations to ensure GPU nodes are properly utilized—all require careful design. The project specifically covers GPU scheduling, indicating the author has indeed encountered related pitfalls in practice.

Technical Details of GPU Scheduling in Kubernetes

Kubernetes treats GPUs as extended resources, indivisible by default—a Pod requesting 1 GPU exclusively occupies that GPU's entire compute power and memory. This is fundamentally different from CPU (divisible by millicores) and memory (divisible by bytes). Several key configurations need attention in actual deployment: First, use taints and tolerations to ensure only GPU workloads are scheduled onto expensive GPU nodes, preventing ordinary Pods from wasting GPU resources; Second, use nodeAffinity to specify model deployment on nodes with specific GPU models (e.g., large models need A100 80GB rather than T4 16GB); Third, for multi-GPU inference (tensor parallelism), all GPUs must be allocated on the same node, achievable through topology constraints. Additionally, NVIDIA's recent MIG (Multi-Instance GPU) and GPU Time-Slicing technologies allow logical GPU partitioning to some extent, suitable for scenarios where small models share the same GPU, but LLM inference with 7B+ parameters typically still requires exclusive use of entire cards.

Deployment Troubleshooting

The author mentions the project includes content on "deployment issues." In real deployments, common problems include: oversized images causing pull timeouts, slow model weight downloads, out-of-memory (OOM) crashes, and driver version mismatches with CUDA versions. This type of experience is rarely detailed in official documentation but is exactly the practical knowledge engineers need most.

Model Weight Loading and Image Optimization Strategies

LLM model weights typically range from several GB to tens of GB (e.g., LLaMA 2 7B is approximately 13GB, 70B approximately 130GB), presenting unique challenges for containerized deployment. There are three common model loading strategies: (1) Baking the model into the container image—simple and direct but results in enormous images, requiring rebuild and push for every model update; (2) Downloading from object storage (such as Azure Blob Storage) at Pod startup—flexible but slow cold start, depending on network bandwidth; (3) Pre-mounting model files using PersistentVolumes—providing shared storage via Azure Files or Azure NetApp Files, allowing multiple Pods to simultaneously mount the same model data in read-only mode, avoiding both image bloat and redundant downloads. In the AKS environment, you can also leverage Ephemeral OS Disks and image caching (through Azure Container Registry's artifact streaming feature) to accelerate large image pulls, reducing what would normally take 10-15 minutes to just 2-3 minutes.

Cloud GPU Cost Control Strategies

This is perhaps the most easily underestimated aspect of self-hosting LLMs. GPU instances (especially high-end cards like A100 and H100) have extremely high cloud unit prices—if a cluster runs idle 24/7, costs can spiral out of control quickly. The project covers cost controls, likely involving strategies such as: using Azure Spot instances to reduce compute costs, configuring Cluster Autoscaler to scale down during off-peak hours, and optimizing batch configurations to improve per-GPU throughput, thereby reducing the total number of GPUs needed.

Azure Spot Instances and GPU Cost Optimization

Azure Spot VMs utilize idle compute capacity in Azure data centers, offering GPU instances at discounts of up to 90% off regular prices. The trade-off is that Azure may reclaim instances at any time with just 30 seconds notice (eviction). For LLM inference services, this presents unique engineering challenges: inference requests are stateless short-duration tasks, so a single eviction only affects a few in-progress requests, and this can be effectively managed through multi-replica deployments and load balancing. A typical hybrid strategy is: maintaining 1-2 on-demand GPU nodes as baseline capacity to guarantee SLA, while using Spot node pools to handle burst traffic. AKS natively supports Spot node pool configuration, and combined with Pod Disruption Budgets (PDB), you can control the number of Pods evicted simultaneously, ensuring the service never goes completely offline. In real-world cases, NC_A100_v4 series Spot instance prices can drop from approximately $3.67/hour (on-demand) to $0.37-1.10/hour, representing very significant cost savings.

Key Considerations for Self-Hosting LLM Decisions

While projects like this provide a clear path for technical implementation, teams should rationally evaluate the following dimensions before making a decision.

Cost Breakeven Point: Self-hosting isn't always cheaper. Only when call volume is sufficiently large and GPU utilization is sufficiently high can self-hosting potentially beat pay-per-use APIs. For sporadic usage scenarios, third-party APIs are often more economical. A rough rule of thumb: when a team's monthly API spending consistently exceeds the monthly cost of a GPU instance (including amortized operations personnel costs), it's worth seriously evaluating self-hosting. Taking GPT-4 level models as an example, if monthly API costs exceed $5,000-10,000, self-hosting an A100 instance (approximately $2,600/month on-demand, potentially as low as $800/month with Spot) paired with open-source models may already be economically viable.

Operational Complexity: From the technical points covered in this project—vLLM, GPU Operator, K8s scheduling, cost optimization—it's clear that the operations threshold for self-hosting is not low. Teams need Kubernetes and GPU operations expertise; otherwise, they risk falling into a long-term maintenance quagmire.

Data Compliance and Sovereignty Value: For industries with strict data sovereignty requirements—finance, healthcare, government—the data controllability that self-hosting provides is often the decisive factor, with cost becoming a secondary consideration. Under regulatory frameworks like the EU's GDPR or China's Data Security Law, sending user conversation data to third-party APIs may pose compliance risks, while self-hosted models ensure data never leaves organizational boundaries, with audit trails that are clearer and more controllable.

Conclusion

The vllm-serving-aks project shared by shiqs90 is a pragmatic engineering reference for LLM self-hosting. Rather than stopping at the "Hello World" level, it comprehensively covers production-essential components including vLLM inference engine configuration, NVIDIA GPU Operator deployment, K8s GPU scheduling strategies, deployment troubleshooting experience, and cost control solutions. For teams evaluating private deployment paths, open-source practices like this provide both a technical blueprint and help identify key risk points in the implementation process. As LLM inference services increasingly move toward enterprise private deployment, mastering this vLLM + Kubernetes + GPU skill stack will become ever more important.