GPU Sharing in Kubernetes Production: A Deep Dive into MIG, MPS, and Time-Slicing

A Classic GPU Resource Waste Scenario

Imagine this: you have a 7B-parameter model that requires only about 14GB of VRAM when running at FP16 precision. The math is straightforward — FP16 (half-precision float) uses 2 bytes per parameter, so 7×10⁹ parameters × 2 bytes = 14GB. That figure doesn't even account for KV Cache, activations, and intermediate computations during inference, which push real-world usage even higher. Yet when you deploy it to a Kubernetes cluster, the scheduler hands it an entire A100 with 80GB of VRAM.

Kubernetes discovers GPU resources through the Device Plugin framework. NVIDIA's k8s-device-plugin registers each physical GPU as a single, indivisible resource unit (nvidia.com/gpu: 1), and the scheduler treats this as its atomic allocation unit. This design simplifies implementation but completely sacrifices fine-grained scheduling. The device plugin reports the GPU as fully occupied, the next Pod gets stuck in Pending — and 66GB of VRAM on that expensive GPU sits completely idle.

This is a recurring pain point in production environments. Kubernetes' default GPU scheduling granularity is "one full card" — it has no awareness of each workload's actual VRAM or compute requirements. For A100s and H100s that cost tens of thousands of dollars each, this coarse-grained allocation translates to massive cost waste.

Fortunately, NVIDIA offers three GPU sharing solutions: Time-slicing, MPS (Multi-Process Service), and MIG (Multi-Instance GPU). The question isn't which one is "best" — it's which one fits your specific scenario.

The Core Decision Framework: Trust Boundaries Between Workloads

The key to choosing a Kubernetes GPU sharing solution comes down to one thing: the trust boundary between workloads.

In other words, you need to answer one question first: do the workloads sharing a GPU require fault isolation from each other? Are they dev/test jobs from the same team, or production services from different teams that have no trust relationship?

Your answer directly determines which technical path to take. Let's break down the applicable scenarios and technical trade-offs for each approach.

Technical Trade-offs of the Three GPU Sharing Solutions

Time-Slicing: Easiest to Get Started, Highest Risk

GPU time-slicing lets multiple workloads take turns using the same GPU's compute resources, achieving time-division multiplexing through context switching. It's worth noting that GPU context switching is fundamentally different from CPU context switching: a CPU context switch saves only a small amount of register state (typically a few KB), whereas a GPU context switch must save the entire GPU state — including all register files, shared memory, and L1 cache contents — which can amount to hundreds of MB. This means frequent context switching introduces significant scheduling latency, and performance degradation is especially pronounced under high concurrency.

Its core advantage is universality and ease of deployment: nearly all NVIDIA GPUs support it, including older T4s and V100s, and configuration overhead is minimal. If all your workloads belong to the same team, are in dev/test, and your only goal is to stop Pods from getting stuck in Pending, time-slicing is usually sufficient.

Its fatal flaw, however, is zero memory isolation and zero fault isolation. All processes sharing a GPU run in the same memory space — a single bad CUDA call can crash every workload on that GPU simultaneously.

MPS (Multi-Process Service): Better Concurrency for Trusted Workloads

MPS goes a step further than time-slicing. It proxies GPU requests from all client processes through a shared CUDA context, allowing CUDA kernels from multiple processes to execute concurrently within the same time slice — fully utilizing the GPU's SM (Streaming Multiprocessor) resources rather than relying on context switching for round-robin exclusive access. This architecture enables truly parallel execution of CUDA kernels across multiple processes, yielding higher concurrent utilization. MPS also supports per-process VRAM limits via the CUDA_MPS_PINNED_DEVICE_MEM_LIMIT environment variable, providing soft resource quota management.

If you're running a set of mutually trusted workloads on an A100 and want to squeeze out higher concurrent performance, MPS is worth considering.

That said, MPS still provides no fault isolation. Because all processes depend on the MPS daemon for coordination, a crash in one process can propagate through the MPS daemon and bring down all other workloads on the same GPU — this is an inherent architectural risk of MPS.

MIG (Multi-Instance GPU): Hardware-Level Isolation, True Multi-Tenancy

MIG is the only solution among the three that provides true hardware-level isolation. It leverages the GPC (Graphics Processing Cluster) physical partitioning capability introduced in the Ampere architecture to divide the GPU's compute units, L2 cache, and memory bandwidth into independent partitions at the hardware level. An A100 80GB can be partitioned into up to seven 1g.10gb instances (approximately 10GB VRAM each), or into larger 2g, 3g, 4g, or 7g profile instances. Each MIG instance has its own dedicated VRAM address space and independent SM resources — a crash in one instance has zero impact on the others. This is a hardware-level guarantee that neither time-slicing nor MPS can offer.

When different teams need to share a GPU and mutual isolation is required, MIG is almost always the right choice.

That said, MIG has clear limitations:

• Only supported on Ampere and newer architectures (A100, H100) — T4 and V100 are not supported;
• Fixed partition profiles — you can only divide using predefined profiles, with no free-form specification;
• Reconfiguration requires draining existing workloads — live dynamic reconfiguration is not supported.

A Practical GPU Sharing Decision Tree

Here is a battle-tested decision flow that is logical and directly applicable to your selection process:

Step 1: Do the workloads require fault isolation from each other?

• No → Choose between time-slicing or MPS based on your concurrency needs.
• Yes → Proceed to Step 2.

Step 2: Does the GPU support MIG?

• Yes (A100, H100) → Use MIG for hardware-level isolation.
• No (T4, V100) → Either accept the risks of shared access, or allocate a dedicated GPU per workload.

The value of this decision tree is that it reduces what looks like a complex technical selection problem into two direct yes/no questions.

The Most Common Selection Mistake: Defaulting to Time-Slicing

In real production environments, the most common mistake is: using time-slicing everywhere simply because it's the easiest to configure.

Time-slicing is easy to adopt — until the day one team's CUDA error takes down three other teams' services and you have to explain to everyone what happened.

In production, configuration convenience should never trump isolation. Especially in multi-team, multi-tenant shared environments, solutions that lack fault isolation are a ticking time bomb. Choosing a Kubernetes GPU sharing solution is fundamentally an engineering decision about risk management, not merely an optimization for resource utilization.

Summary: Define the Boundary First, Then Choose the Solution

GPU is one of the most expensive resources in AI infrastructure, and its utilization rate directly affects cost efficiency. But blindly chasing utilization while ignoring isolation often leads to far greater costs in production.

The right selection logic is: first clarify the trust boundaries between workloads, then choose accordingly —

• Same-team dev/test: use time-slicing for configuration simplicity;
• Trusted workloads with high concurrency demands: use MPS to maximize concurrent performance;
• Multi-team, multi-tenant production environments: MIG is mandatory for hardware-level isolation.

Understanding the trade-offs and boundaries of each solution matters more than searching for a single "optimal answer." So how is your team currently handling GPU sharing — assigning dedicated GPUs to each Pod for simplicity, or actually doing partitioned multiplexing?