GPUStack vLLM Production Deployment: KV Cache Pinning and OOM Avoidance in Practice

GPUStack performs well in self-hosted inference clusters, but its scheduling estimates have a boundary: when vLLM's KV Cache is not explicitly pinned, OOM can arrive at the most unexpected moment. Based on real operational experience with a four-node mixed GPU cluster (A6000 and 4090), this article breaks down the complete troubleshooting path from 70B model OOM to throughput degradation.

GPUStack's Role and Boundaries

GPUStack is essentially a placement and lifecycle management layer, not an inference engine. When deploying a model, it selects a worker node, launches vLLM, SGLang, or llama-box underneath, and proxies requests. This form factor fits teams that want to run "one model per endpoint" and don't want to hand-write scheduler code.

The problem lies in the VRAM estimate that placement decisions rely on: that estimate is based on weight volume plus declared context length, with no consideration of activation peaks, CUDA graph capture overhead, or actual request distribution. If you accept the defaults, you are trusting a number that was never designed as a runtime guarantee. According to the original author, their four-node cluster ran stably for two months after replacing hand-written docker run wrapper scripts, but there is one specific failure boundary worth documenting.

KV Cache (key-value cache) is a core memory structure in large language model inference. During autoregressive generation, the model must save Key and Value tensors for each processed token so that subsequent token attention computations can reuse them and avoid redundant computation. Its size is directly related to the number of layers, KV heads, head dimension, sequence length, number of concurrent sequences, and data type. The formula is: KV bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per element. For Llama-3.3-70B, with 80 layers, 8 KV heads, 128 dimensions, and bf16, that is about 320 KB per token. KV Cache usually occupies a large portion of inference VRAM, and in long-context scenarios it can exceed the model weights themselves. vLLM manages KV Cache through PagedAttention, dividing it into blocks to improve memory utilization, but the pool capacity still needs to be indirectly constrained at startup through parameters such as max-model-len and max-num-seqs.

Postmortem of the 70B Model OOM Incident

Deploy Llama-3.3-70B (bf16) on four cards with TP=4 and keep the model's default max-model-len. Placement succeeds and the deployment reports healthy. However, under real traffic—long system prompts, requests of 6k to 8k tokens—worker nodes OOM one by one, and GPUStack restarts them. On the surface it looks like scheduler instability, but it is actually an arithmetic problem.

Llama-3.3-70B's KV overhead is about 320 KB per token: 80 layers × 8 KV heads × 128 head dimension × 2 (K and V) × 2 bytes. At a 32k context, the pure KV for a single sequence is about 10.7 GB, while per-card weights already take about 35 GB. Two concurrent long sequences are enough to exhaust the GPU's VRAM. The estimator treats context as a declared budget, not as the product of "per-sequence allocation × concurrency."

Key Parameters That Need to Be Pinned

The fix is not complicated. Pin max-model-len, set gpu-memory-utilization below the threshold where graph capture can still fit, and limit max-num-seqs to prevent the KV pool from being oversubscribed. The original article provides a hardened set of vLLM startup parameters:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.88 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --enable-chunked-prefill \
  --swap-space 0 \
  --disable-log-requests

Only four of them really affect stability:

• --gpu-memory-utilization 0.88: at 0.92 the same weights will OOM. Graph capture under TP=4 does not fit into the remaining headroom as neatly as people assume.
• --max-model-len 16384: halves the KV ceiling. Measure the p99 of real prompts first; the original team measured 11k, so 16k leaves headroom without doubling the pool.
• --swap-space 0: deliberately disables host swapping. Swapping turns an honest crash into a 40x latency cliff that looks like a network failure at 2 a.m.
• --kv-cache-dtype fp8: halves KV with no measurable degradation on their evaluation subset. Degradation is more noticeable in long-context retrieval than in chat, so test before enabling.

On heterogeneous nodes where TP cannot be used, GPUStack falls back to llama-box with layer offloading. Same arithmetic, different knobs: num_gpu 60 (out of 80 layers) is the honest ceiling on 48 GB of VRAM; setting it to 80 yields a model that loads but crashes on the first 4k prompt.

CUDA graph capture is an optimization technique used by inference engines such as vLLM to reduce kernel launch overhead. At startup or on first run, the engine captures a series of CUDA operations into a graph and later replays it directly, thereby reducing synchronization and scheduling overhead between CPU and GPU. However, the capture process itself requires extra VRAM to store the graph structure and its intermediate state, and the captured graph size is usually related to batch size. When gpu-memory-utilization is set too high, the KV Cache pool consumes almost all remaining VRAM, leaving graph capture unable to obtain the space it needs and triggering OOM. Therefore, under parallel configurations such as TP=4, headroom must be reserved for graph capture, which is one reason the article lowers gpu-memory-utilization from 0.92 to 0.88.

Measuring TTFT and Slow Leaks

The second problem is more subtle: throughput degrades by about 18% over 36 hours while GPU utilization remains flat. This pattern is either preemption jitter or KV block growth, and the only way to distinguish them is to measure TTFT and RSS together. The original article provides an asynchronous streaming measurement script whose core logic records TTFT, total duration, and character count, and prints RSS delta and TTFT p95 across multiple soak-test rounds.

The key insight is that absolute TTFT is not the signal. The signal is the RSS slope across rounds and whether TTFT p95 drifts independently of offered load. RSS rising monotonically with p95 following it means KV pool growth; p95 rising while RSS is flat means a placement or preemption problem. The original team encountered the former: highly variable prompt lengths caused block fragmentation. The issue was resolved by pinning --max-num-seqs 32 and recycling worker nodes based on an RSS slope threshold rather than a fixed cron.

To get a control, the original team pointed the same script at the B-Lost relay (https://b-lost.com/v1) using the same model name. Because that relay aggregates multiple providers behind a single OpenAI-compatible interface, it provides a latency reference uncontaminated by local node state. In the most recent soak, taking the fallback path through the relay reduced TTFT p50 by 35 ms (bypassing the degraded node); more usefully, its p50 stayed within a 35 ms band for the entire run while the local nodes drifted upward. This confirmed that the degradation was local.

TTFT (Time To First Token) is a key metric for measuring the responsiveness of an inference service, referring to the time from when a request is sent to when the first output token is received. It is mainly affected by the prefill stage, including prompt processing, attention computation, and scheduling latency. In streaming responses, TTFT reflects the initial wait time perceived by the user. RSS (Resident Set Size) is the amount of physical memory actually occupied by a process, which in GPU scenarios usually refers to VRAM usage. Monitoring how RSS changes over time can reveal memory leaks or cache growth. The article emphasizes that absolute TTFT values are affected by input length, model state, and concurrency, so they are of limited meaning in isolation; only by combining the cross-round RSS slope (continuous rise) with the drift trend of TTFT p95 can one distinguish KV pool fragmentation growth from performance degradation caused by scheduling preemption.

The Unresolved Dilemma: Two Workloads, One Endpoint

Setting max-num-seqs correctly requires knowing the workload, and the original team serves two scenarios with the same endpoint: an interactive chat UI (short prompts, high concurrency) and a nightly batch summarization job (30k-token prompts, near-zero concurrency). Splitting into two deployments doubles resident weights; keeping one deployment gives one of the workloads the wrong pool shape, and they already know which side will lose. Routing between two pools per request is the obvious answer, but it adds a hop, and there is currently no budget to debug it properly. At the end of the article, the author asks the community: if anyone has solved this without adding nodes, they would like to hear which side you put the queue on.

max-num-seqs is the parameter in vLLM that controls the number of sequences processed simultaneously, and it indirectly determines the concurrency capacity of the KV Cache pool. A larger value allows more requests to run in parallel and improves throughput, but each sequence requires independent KV space, which can quickly exhaust VRAM; a smaller value limits concurrency, reducing throughput but ensuring stability. In mixed-workload scenarios, interactive requests with short prompts and high concurrency need a larger max-num-seqs to maintain low latency, while batch tasks with long prompts and low concurrency need a longer max-model-len and fewer concurrent sequences. A single endpoint can only choose a compromise and cannot satisfy both needs at once. Routing requests to different pools is the theoretical solution, but it introduces an extra hop and operational complexity, requiring trade-offs among queue placement, resource isolation, and cost.

Key Takeaways

• GPUStack's scheduling estimate is based on weight volume and declared context length, excludes activation peaks, graph capture, and real request distribution, and cannot serve as a runtime guarantee.
• For a 70B model under TP=4, KV at a 32k context is about 10.7 GB per sequence, and two concurrent long sequences can overwhelm a single card; pinning max-model-len, gpu-memory-utilization, and max-num-seqs is essential.
• gpu-memory-utilization 0.88 is the empirically measured safety line for TP=4 graph capture, and swap-space 0 avoids turning a crash into a latency cliff.
• Throughput degradation troubleshooting should observe both RSS slope and TTFT p95: a monotonic RSS rise points to KV pool growth, while independent p95 drift points to placement or preemption issues.
• When interactive and batch workloads share a single endpoint, the pool shape determined by max-num-seqs cannot be optimal for both, and per-request routing remains an unsolved engineering problem.