Ollama Crawling on K3s? A Practical Guide to GPU Passthrough Configuration

Title

Ollama Crawling on K3s? A Practical Guide to GPU Passthrough Configuration

Meta Description

Ollama inferencing slowly on K3s? The root cause is that containers can't see the NVIDIA GPU, falling back to CPU by default. This article breaks down the reasons and provides a complete GPU passthrough configuration: NVIDIA Container Toolkit, containerd template, Device Plugin deployment, and Ollama resource declaration, with verification methods and troubleshooting. Speed goes from 1 token per second to 60-90 tokens.

Article Content

Deploying Ollama on K3s to run local large models should be a win-win: privacy-preserving and cost-saving. But many people follow tutorials and find generation speed is only 1 token per second, CPU fans spinning wildly, memory maxed out—the experience collapses. The problem isn't the model, nor K3s itself, but that containers by default can't see your NVIDIA GPU at all. This guide breaks down the reasons and provides a complete GPU passthrough configuration path.

Why Ollama Runs on CPU by Default in K3s

On bare-metal macOS or Windows with CUDA drivers, Ollama can automatically detect the GPU and offload neural network layers to VRAM. But Kubernetes is a deliberately isolated containerized environment, and K3s's built-in containerd runtime by default does not expose the host's PCI hardware to containers.

The result is a chain reaction: the Ollama process inside the container can't see /dev/nvidia* device nodes, can't load the CUDA runtime, and quietly falls back to CPU for matrix multiplication. Inferencing 7B or 13B models on CPU involves continuous high-latency data movement between system memory and CPU cache, ultimately manifesting as thread contention, thermal throttling, and generation speeds far below GPU VRAM solutions.

The fix is to build a hardware bridge from the Linux kernel driver to the K3s Pod namespace. The entire chain has four layers: Linux kernel and NVIDIA driver → NVIDIA Container Toolkit and runtime → NVIDIA K8s Device Plugin → Ollama Pod with GPU resource declaration.

containerd and OCI Runtime: K3s uses embedded containerd as its container runtime by default, and containerd starts containers via the OCI (Open Container Initiative) runtime specification. The default runc runtime only handles basic isolation like creating namespaces and mounting filesystems; it does not actively mount host device nodes (like /dev/nvidia*) into the container. To give containers GPU access, you must switch to a runtime that can inject devices (like nvidia-container-runtime), or enable the NVIDIA runtime plugin in the containerd configuration. This is why even with a healthy host driver, Ollama inside the container still can't see the GPU—the OCI runtime simply doesn't expose the device.

Step 1: Install Drivers and Container Toolkit on the Host

Before modifying K3s, confirm the host itself is healthy. Run nvidia-smi; you should see GPU model, driver version, and CUDA version. If this step fails, subsequent configuration is meaningless.

Next, install the NVIDIA Container Toolkit, which enables containerd to inject host GPU devices into OCI container specs. On Ubuntu/Debian, install via the official repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb [^ ]* #&[signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] #' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Step 2: Make K3s's containerd Recognize the NVIDIA Runtime

K3s uses its own embedded containerd, with configuration at /var/lib/rancher/k3s/agent/etc/containerd/config.toml. Here's a critical pitfall: editing this file directly won't work—K3s overwrites it on every restart. The correct approach is to use the template extension mechanism.

First, copy the existing configuration to a .tmpl template file, then register nvidia-container-runtime in the template:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

After editing, run sudo systemctl restart k3s to apply. Only after this step does containerd have the ability to start CUDA containers.

Step 3: Deploy the Device Plugin So the Scheduler Sees the GPU

containerd being able to run CUDA containers isn't enough; the Kubernetes scheduler needs to know how many GPUs are available for allocation on the node. This is handled by the NVIDIA Kubernetes Device Plugin, installed via Helm into the kube-system namespace with runtimeClassName=nvidia specified.

After deployment, check with kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'. If the result contains "nvidia.com/gpu": "1", the cluster has GPU awareness and the scheduler can treat the GPU as an allocatable resource.

Kubernetes Device Plugin Mechanism: Device Plugin is a framework Kubernetes uses to extend schedulable resources (like GPUs, FPGAs, NICs). It registers resource names (e.g., nvidia.com/gpu) and quantities with kubelet via gRPC, and kubelet reports these to the API Server, ultimately reflected in the node's allocatable field. The scheduler relies on this information to decide if a Pod can be allocated a GPU. The NVIDIA Device Plugin runs as a DaemonSet on each node, discovering GPUs and managing container device allocation. Without it, the cluster doesn't know the node has a GPU, and Pods requesting nvidia.com/gpu stay Pending forever.

Step 4: Declare GPU Resources for Ollama

This is the step where you see the results. When deploying Ollama, you must explicitly declare GPU requests and limits in the Pod spec, and specify nvidia in runtimeClassName, otherwise all previous configuration is wasted. The core snippet:

spec:
  runtimeClassName: nvidia
  containers:
  - name: ollama
    image: ollama/ollama:latest
    resources:
      requests:
        nvidia.com/gpu: "1"
      limits:
        nvidia.com/gpu: "1"

Note that both requests and limits must include nvidia.com/gpu. Specifying only requests without limits can cause resource allocation anomalies in some scheduling scenarios.

The Role of runtimeClassName: Setting runtimeClassName: nvidia in the Pod spec tells kubelet to use the RuntimeClass named nvidia to run the Pod. RuntimeClass is a Kubernetes resource that associates a Pod with a specific container runtime configuration. The nvidia runtime referenced here is exactly the nvidia-container-runtime registered via the containerd template in Step 2. Without specifying runtimeClassName, even with GPU resources declared, the container still starts with the default runc runtime, preventing device injection. Therefore, resource declaration and runtimeClassName must both be configured—neither can be omitted.

Verification: Confirm the GPU Is Actually Working

After deployment, run ollama run llama3:8b inside the Pod, and simultaneously run watch -n 1 nvidia-smi in another terminal on the host. Three signals confirm successful passthrough: ollama_llama_server appears in the process list, VRAM usage jumps from 0 to about 4.5–5.5 GiB (for a quantized 8B model), and generation speed reaches 60–90 tokens per second. Getting these numbers means the entire chain is working.

Three Common Failures and Solutions

Pod stuck in Pending with "Insufficient nvidia.com/gpu". This means the Device Plugin didn't register successfully. First confirm nvidia-smi works on the host, then restart the daemonset: kubectl rollout restart daemonset nvidia-device-plugin-daemonset -n kube-system.

CUDA out of memory when loading the model. The model parameter count exceeds VRAM capacity. Switch to a smaller quantized version, e.g., use llama3:8b-instruct-q4_K_M instead of FP16, or limit parallel requests via OLLAMA_NUM_PARALLEL=1.

GPU passthrough suddenly fails after a reboot. Most likely a Linux kernel auto-upgrade caused a mismatch with the NVIDIA DKMS module. Disable automatic updates for the kernel and NVIDIA packages, or pin the driver version, e.g., sudo apt-mark hold nvidia-driver-550.

DKMS and Kernel Modules: NVIDIA drivers typically use DKMS (Dynamic Kernel Module Support) to automatically recompile kernel modules on each kernel update. If DKMS compilation fails or isn't triggered after a kernel upgrade, nvidia-smi may report "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver", causing GPU passthrough to fail. Pinning the driver version (e.g., apt-mark hold) avoids mismatches from automatic upgrades, but in the long run you still need to manually verify DKMS status (dkms status) and rebuild modules after kernel upgrades.

Is This Configuration Worth It?

The value of running Ollama on K3s lies in combining lightweight orchestration with local GPU compute to form a controllable private AI infrastructure. The configuration cost of GPU passthrough is concentrated in the initial four-layer bridge. Once established, subsequent deployments—vector databases for RAG, K8sGPT for cluster diagnostics, Longhorn for persistent storage—all build on the same GPU-aware cluster. Latency drops from 30-second levels to milliseconds, a very worthwhile ROI for teams with data compliance requirements.

If you only need single-machine inference and don't plan to scale to a multi-service architecture, running Ollama directly on the host is actually simpler. The K3s approach suits scenarios where services are already containerized and require unified scheduling and resource isolation. Understanding your territory before acting can save you a lot of hassle.

One-line Summary

Detailed explanation of why Ollama defaults to CPU on K3s and the four-layer GPU passthrough configuration, with verification and troubleshooting.

Paragraph Summary

The article addresses the issue of GPU not being utilized and extremely slow generation when deploying Ollama on K3s, pointing out the root cause is that K3s's embedded containerd by default isolates host PCI hardware, so containers can't see NVIDIA device nodes, causing Ollama to fall back to CPU inference. The solution is to build a four-layer hardware bridge: install NVIDIA drivers and Container Toolkit on the host, register the NVIDIA runtime in K3s's containerd via template extension, deploy the NVIDIA K8s Device Plugin so the scheduler is GPU-aware, and declare GPU resources and specify runtimeClassName in the Ollama Pod. The article also provides three signals to verify successful passthrough (process, VRAM usage, generation speed), and offers solutions for three types of failures: Pod Pending, CUDA OOM, and driver failure due to kernel upgrades, finally discussing applicable scenarios and ROI.