Who's Eating My GPU? Caller-Level Attribution Reveals the Truth About a Homelab RTX 3090

From "GPU at 100%" to "Who's Using the GPU"

Anyone who has built a homelab has run into this scenario: GPU utilization spikes to 100%, fans spin up like a jet engine, and you have absolutely no idea which of the twenty-odd services on the machine is the culprit. You open nvidia-smi, it shows one process, you close the window — and the mystery remains.

The core problem is that nvidia-smi answers the question "which process is using the GPU." In a Docker-based stack, that process is usually the container runtime, not the service actually doing the work. What you really want to know is: "which service is using the GPU, for how long, and who initiated the call?"

This hands-on article — part 10 of the "The Local LLM Lab" series — describes how the author added a critical capability to their homelab monitoring system: caller→server attribution. Every GPU utilization sample is tagged with the service that owns the process, and for shared services (such as transcription and embedding endpoints), the caller that initiated the request. Over 7 days, the attribution system recorded 1,055 samples and roughly 1.1 million seconds of GPU time. The findings are revealing.

Experimental Setup: One RTX 3090, Two Shared Services

The entire environment revolves around a single NVIDIA RTX 3090 (24GB VRAM, power limit 390W). Two shared services run on the card:

• Whisper ASR transcription service — fronted by an nginx proxy, receiving requests from the author's various tools, observability stack, CI, and a warmup job.
• nomic-embed-text (Ollama) — an embedding endpoint used by a RAG pipeline.

The most important design decision in the whole setup: caller information comes from the request itself, not from the process. When the author's assistant core sends an audio clip to the whisper proxy, that sample is tagged as jarvis2-core → whisper-proxy, not nginx → whisper. This is the essential difference between "the GPU is busy" and "my assistant is using it."

Whisper ASR is OpenAI's open-source automatic speech recognition model, supporting multilingual transcription and translation. Its combination of high accuracy and local deployability makes it popular in homelab environments. It's a compute-intensive inference workload — each transcription briefly maxes out the GPU, making it a classic source of utilization spikes. nomic-embed-text is a text embedding model released by Nomic AI, designed to convert text into vector representations, and is commonly used in the semantic retrieval stage of RAG (Retrieval-Augmented Generation) pipelines. Both services are "shared endpoints" — multiple upstream callers send requests concurrently or in alternation. This is precisely why nvidia-smi alone can't identify the true source of consumption: the process is always the same, but the request originators behind it vary.

That Number: 47%

Over 7 days, 47% of all attributable GPU time flowed to a single caller — the author's own assistant core (4,970 seconds out of 10,555 total).

Caller	Seconds (7 days)	Server	Share
jarvis2-core (assistant core)	4,970	whisper-proxy	47.1%
whisper-proxy (direct calls)	920	whisper service	8.7%
langfuse worker (observability)	910	whisper service	8.6%
langfuse minio (object storage)	860	whisper service	8.1%
whisper-warmup (keepalive job)	360	whisper service	3.4%
gh-runner (CI)	360	whisper-proxy	3.4%
jarvis-server + watchtower	410	whisper-proxy/service	3.9%
firefly importer	80	whisper service	0.8%
ollama-embed callers	210	ollama-embed	2.0%
UptimeKuma (health checks)	20	whisper service	0.2%

The GPU's biggest consumer is neither a large language model nor a benchmark or training job — it's the assistant transcribing audio it recorded itself. Voice memos, meeting recordings, and audio attached to traces collected by the observability stack all flow through the Whisper endpoint and get attributed to their respective originators.

What Attribution Changes

Without caller attribution, the entire 7-day story for this GPU is a single sentence: "The Whisper service used 10,555 seconds of GPU time." That's just a number — it doesn't guide any action.

With caller attribution, the story becomes three-dimensional:

The assistant is the top tenant. 47% of GPU time was consumed by the author's own toolchain — the service was built for it, and its usage exceeds all other callers combined. This confirms the design is working as intended, and explains why this service needs to run 24/7 rather than on demand.

The observability stack is the second-largest tenant. Langfuse's worker and MinIO together account for 17% of GPU time — they're transcribing audio attached to stored LLM traces. This is a cost nobody budgeted for: the tracing stack itself is doing GPU work, layered on top of the LLMs it's observing.

CI is a real but small tenant. The GitHub runner accounts for 3.4% — test fixtures being transcribed during CI runs. Not a large number, but