Running Ollama and ComfyUI Simultaneously on a Single RTX GPU Without OOM: A Practical Guide

In a home lab setup, many people want to run Ollama for local LLM inference as a coding assistant while also using ComfyUI to generate Stable Diffusion images. The problem is that most people only have a single 16GB consumer GPU (like the RTX 5060 Ti), and having two VRAM-hungry applications competing for resources makes OOM (Out of Memory) crashes almost inevitable. This guide draws on real configuration experience to break down how to elegantly serve both workloads from a single GPU.

First, Do the Math: How Much Can 16GB Actually Hold

Before adopting any sharing strategy, you need to understand exactly how much VRAM each application actually consumes. Blindly launching both will only result in a steady stream of crash logs.

Ollama VRAM Usage (After Quantization)

Using Q4_K_M quantization as a reference:

• Qwen2.5-Coder:14b / DeepSeek^Ad-R1:14b: ~7.5GB
• Llama3.1:8b / Qwen3:8b: ~4.5GB
• Nomic-Embed-Text embedding model: ~1.2GB
• System overhead + CUDA context: ~1–2GB

Quantization is a technique that compresses model weights from high-precision floating-point representations (e.g., FP16/FP32) into low-bit integer formats. In Q4_K_M, "Q4" means each weight is stored as a 4-bit integer (roughly 75% smaller than FP16's 16 bits), "K" denotes the K-quants grouped quantization algorithm (which optimizes precision per layer), and "M" indicates a medium precision tier (between Small and Large). This naming convention comes from the llama.cpp project's GGUF format — and since Ollama runs on top of llama.cpp, it supports this system natively. Quantization inevitably introduces some precision loss, but for most everyday tasks, the perceptual difference between Q4_K_M and FP16 output is negligible, while the VRAM savings are substantial — which is exactly why it's the go-to choice for home labs.

ComfyUI VRAM Usage

The Stable Diffusion side has more variables:

• ComfyUI base + VAE: ~1.5–2GB
• SDXL (1024×1024): ~6–8GB
• SD 1.5 (512×512): ~3–4GB
• Each ControlNet unit: +1–2GB
• Hi-Res fix + upscaling: +2–4GB

Which Combinations Are Actually Viable

Adding these numbers together lets you predict success or failure before you even launch:

• Ollama 14b + SDXL 1024×1024 ≈ 14.5GB (barely feasible, very tight)
• Ollama 14b + SD 1.5 + ControlNet ≈ 13.5GB (comfortable range)
• Two 14b models ≈ 15GB (only 1GB left for the system — not recommended)
• Ollama 14b + SDXL + Hi-Res + ControlNet: almost certainly OOM

The real value of this table is that it turns the vague question of "can they run together" into a concrete addition problem you can solve in advance.

Scheduling Strategy: Time-Based Slots vs. Priority-Based Preemption

VRAM is a physical constraint — you can't conjure more out of thin air. The core idea is to make both applications use the GPU in an "off-peak" or "tiered" fashion.

Strategy 1: Time-Based Scheduling (Shift System)

Assign each application a dedicated time window. For example: work hours (6:00–10:00, 14:00–18:00) are reserved for Ollama as a coding assistant; creative hours (10:00–12:00, 18:00–22:00) are for ComfyUI; and overnight slots go back to Ollama for background embedding computation. You can implement this with systemd timers or cron, paired with gpu-grant / gpu-revoke scripts to switch access.

The upside is predictability and simplicity; the downside is inflexibility — if you suddenly need to look up a code snippet during a creative window, you'll have to wait for the next slot.

Strategy 2: Priority-Based Preemption (Reservation System)

Another approach lets applications actively request GPU access, with an arbiter allocating resources based on priority. A three-tier priority system works well here: interactive user requests (priority 10, such as live chat or image generation), background processing (priority 5, such as embeddings and batch jobs), and maintenance tasks (priority 1).

Each application calls requestAccess before invoking generation and releaseAccess when done; higher-priority requests can preempt lower-priority holders. This design more closely matches real-world usage — interactive tasks are always more urgent than background ones.

This arbiter pattern draws conceptually from operating system resource scheduling. GPUs don't have built-in preemptive schedulers the way CPUs do. NVIDIA's MPS (Multi-Process Service) and MIG (Multi-Instance GPU) offer some level of multi-process sharing, but MIG is only available on data-center GPUs (like the A100 and H100) — consumer RTX cards don't support it. MPS allows multiple CUDA processes to share a GPU but doesn't provide priority-based preemption, and lacks memory isolation guarantees between processes. The arbiter approach described here therefore implements scheduling logic at the application layer using token or lock mechanisms to serialize GPU access, working around the lack of fine-grained scheduling support at the hardware level.

The Recommended Hybrid Approach

In practice, combining both strategies works best: use time-based scheduling to handle predictable workload transitions, use priority preemption to handle sudden interactive requests, let background tasks fill idle periods, and implement "graceful preemption" (save state and resume later) rather than abrupt interruptions.

VRAM Partitioning: Static Split vs. Dynamic Management

Beyond the time dimension, you can also partition VRAM spatially.

Static Partitioning (Fixed Split)

Pre-allocate a VRAM ceiling for each application and enforce hard boundaries. Three typical configurations:

• 7GB/9GB (Ollama-heavy): Ollama runs a 14b model; ComfyUI runs SDXL + single ControlNet. Best for coding-focused development work.
• 5GB/11GB (Balanced): Ollama runs an 8b model; ComfyUI can use multiple ControlNets + Hi-Res fix.
• 3GB/13GB (ComfyUI-heavy): Ollama handles only embeddings or very small models; ComfyUI focuses on batch image generation and video.

The fundamental weakness of static partitioning is that an idle application's quota goes to waste, it can't handle burst loads, adjusting the split requires restarting applications, and correctly implementing it at the CUDA/Vulkan level is genuinely complex.

Dynamic Partitioning (Recommended)

A more pragmatic approach lets applications use available VRAM freely, but sets soft limits and monitors usage. A Python monitoring script built around nvidia-smi works well here: read VRAM usage in real time, prompt at 80%, warn at 90%, and trigger cleanup or notifications near 95%.

For most home lab users, the author's recommended dynamic strategy is: use a smaller 8b model for daily assistance during the day; switch to a 14b model for dedicated LLM sessions; keep ComfyUI on moderate settings and save extreme parameters for dedicated creative sessions; and always keep VRAM monitoring running to avoid surprises.

Quantization: The Highest-Leverage Tool for VRAM Efficiency

Choosing the right quantization level determines whether your configuration "fits" or "chokes." Using a 14b model as an example:

Quantization	VRAM Usage	Quality Impact
Q8_0	~10.5GB	Minimal
Q6_K	~9.0GB	Very subtle
Q5_K_M	~7.8GB	Slight
Q4_K_M	~7.5GB	Noticeable but acceptable
Q3_K_M	~6.2GB	Moderate
Q2_K	~5.0GB	Significant

The practical guidance is clear: use Q4_K_M or Q5_K_M for coding assistance; use Q5_K_M or Q6_K for math and reasoning tasks to preserve output quality; Q3_K_M or even Q2_K is fine for casual chat to save VRAM; always use quantized versions of embedding models; and avoid F16 unless you have 24GB or more. You can even pull multiple quantized versions of the same model and switch between them based on current needs.

ComfyUI Optimization: Squeezing Every Drop from Limited VRAM

The Stable Diffusion side has plenty of tuning knobs:

• Use SD 1.5 instead of SDXL when you can — it cuts VRAM usage roughly in half
• Enable attention slicing to trade compute for memory
• Enable VAE tiling to process in chunks and reduce peak VRAM
• Limit the number of ControlNet units — each one adds 1–2GB
• Keep batch size at 1
• Enable model offloading to move idle components to system RAM
• Use xformers for a more memory-efficient attention implementation

Resolution has a dramatic impact on VRAM too. At SD 1.5, 512×512 needs only ~3.5GB, while 1024×1024 with Hi-Res fix can spike to ~12GB; SDXL is even more demanding, with 1280×720 plus Hi-Res reaching ~15GB — already at the limit on a 16GB card. Launching with --medvram and setting PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 to reduce memory fragmentation are common survival tactics.

Both attention slicing and xformers target the self-attention operation, which is the most computationally expensive part of the Transformer architecture. Standard self-attention has VRAM complexity proportional to the square of sequence length — particularly painful at high image resolutions. Attention slicing splits the attention heads into multiple batches processed sequentially, trading more compute steps for lower peak VRAM. xformers uses fused kernels like Flash Attention to dramatically reduce intermediate tensor memory usage and improve speed, while remaining mathematically equivalent — the two techniques can be used together. VAE tiling works at the image encoding/decoding stage: decoding high-resolution images through the VAE produces enormous intermediate tensors, and processing them in tiles keeps peak VRAM under control at the cost of slightly longer processing time.

Queue Management and Monitoring: Preventing Resource Exhaustion

Even with careful scheduling, concurrent requests can still spike VRAM over the limit. Implement request queues for both Ollama and ComfyUI, with the core principle being max_concurrent=1 serial processing — ensuring only one heavy task occupies the GPU at any moment — with support for priority-based queue jumping.

For monitoring, the key metrics to watch are: VRAM usage (three-level alerts at 80%/90%/95%), GPU utilization, temperature (alert above 80°C to prevent throttling), power draw, application response time, queue depth, and error rate. A monitoring script using nvidia-smi for GPU stats, combined with health checks against Ollama's /api/tags endpoint and ComfyUI's local port, provides solid coverage — trigger an alert immediately if either becomes unresponsive.

Troubleshooting: Handling the Most Common Issues

Frequent OOM crashes: Implement graceful degradation — catch CUDA OOM exceptions and automatically retry with a shorter max_length, accepting lower quality over a crash.

Performance drops during context switches: Usually caused by model reloading and VRAM fragmentation. Mitigate by keeping models warm-loaded, using fast NVMe storage, and maintaining a pool of pre-quantized model variants.

Unresponsive applications: Introduce heartbeat mechanisms, auto-restart after N seconds of no response, and add health check endpoints for each application.

VRAM fragmentation: Schedule regular garbage collection, call torch.cuda.empty_cache() in PyTorch applications, and restart processes when necessary to defragment.

Final Thoughts

The author's core insight is worth keeping in mind: the goal is not to run both systems at full capacity simultaneously — that's simply not realistic on consumer hardware. The real goal is to have both systems available when you need them, with predictable performance and a frustration-free experience.

The recommended starter configuration is: Ollama running Qwen2.5-Coder:14b-q4_K_M or Llama3.1:8b-q4_K_M; ComfyUI with SD 1.5, attention slicing, and VAE tiling enabled; a hybrid scheduling approach combining time slots with priority preemption; VRAM alerts and health checks for monitoring; and batching similar tasks together in your workflow to minimize context-switching overhead.

With careful scheduling, sensible model selection, and basic monitoring, a single RTX 5060 Ti is fully capable of serving as a dual-purpose AI workstation for both development assistance and creative image generation.

Background Notes

VRAM fragmentation is a common problem in GPU memory management: after repeatedly allocating and releasing tensors of varying sizes, free VRAM gets split into many small non-contiguous blocks. Even when total free memory appears sufficient, a single allocation requiring a large contiguous block may fail — triggering OOM. PyTorch's CUDA memory allocator uses a caching pool by default to mitigate this; torch.cuda.empty_cache() returns free blocks from the cache to CUDA (making them visible to other processes), but doesn't reorganize already-allocated memory. PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 limits the maximum granularity at which the allocator splits large memory blocks, reducing fragmentation. In an Ollama + ComfyUI alternating-use scenario, both processes hold cached memory without releasing it — which is a primary driver of fragmentation. Proactively triggering cleanup or restarting processes before switching workloads is therefore a genuinely useful practice.