Running Three AI Services on a 24GB GPU: Real-World Usage Peaked at Only 17.8GB
Three AI services on one 24GB RTX 3090: 17.8GB sustained, 23.4GB peak, zero spill over 7 days.
A week of real VRAM measurements on a single RTX 3090 running Whisper ASR, a TTS model, and nomic-embed-text simultaneously shows 17.8GB sustained usage (72%) and a 23.4GB peak with no memory spill. The key finding: always-resident services consume 1.2–2.5× their weight size due to CUDA context, framework buffers, and KV caches — a 0.3GB embedding model actually uses 4.6GB. The practical takeaway: size your GPU by sustained working set plus 50% headroom, not by peak or spec-sheet weights.
The Truth About VRAM Requirements: Beyond the Spec Sheet
Online advice about "how much VRAM do I need" almost always follows the same playbook: here's a table. "27B at FP8 needs 28GB." "70B at Q4 needs 42GB." "Just buy a 48GB card."
These answers only work for a single model in isolation. Nobody tells you what happens when you pack a real-world stack — a transcription service, an embedding model, a small chat model — onto the same GPU. The answers you find are mostly guesswork, because almost nobody has actually measured a multi-tenant shared GPU.
This article is based on a real-world report from a home AI enthusiast. Over the past week, his RTX 3090 (24GB) ran three AI services simultaneously, with full VRAM history curves and per-tenant attribution data collected. This is what a GPU actually consumes in practice — and a genuine basis for choosing a home GPU.
The Test Setup: A Real Home AI Stack
The test platform is an NVIDIA RTX 3090 with a 390W power limit and 24GB (24,576MB) of VRAM. Three always-resident tenants all run as Docker containers:
- Whisper ASR web service (WhisperX, large-v3) — an endpoint for transcribing voice memos, meetings, and podcast clips
- A small TTS voice model — providing speech synthesis for the same assistant stack
- nomic-embed-text (Ollama) — generating embeddings for a RAG pipeline
The same machine also runs 55 containers, 92 systemd units, a 56-thread Xeon processor, and 64GB of RAM — but the GPU exclusively serves these three tenants.
The author emphasizes that this combination is exactly what a typical home AI stack looks like: one larger always-resident model (ASR), one medium always-resident model (TTS), and one tiny always-resident model (embeddings). No chat model — a 27B chat model runs on a separate machine with dual 3090s, which is a different story.
WhisperX is an enhanced version of OpenAI's Whisper, adding two extra modules on top of the base speech-to-text capability: VAD (Voice Activity Detection) to strip silence before transcription for better efficiency, and a forced alignment model (based on wav2vec2) for word-level precise timestamps. Both add-on models are resident on the GPU, which explains much of the gap between the Whisper large-v3 weights (3.1GB) and the measured average (7.7GB). nomic-embed-text is an open-source text embedding model from Nomic AI with approximately 137 million parameters, designed for RAG (Retrieval-Augmented Generation) scenarios. It encodes text chunks into 768-dimensional vectors with a context window of up to 8,192 tokens, delivering strong retrieval performance at its parameter scale, and can be pulled and run with a single command via Ollama.
The Key Numbers: 17.8GB Resident, 23.4GB Peak
The core finding in a single sentence: with three models always resident, the GPU used 17.8GB (72%), peaked at 23.4GB over the course of a week, and not a single byte spilled to system memory.
| Tenant | 7-Day Average VRAM | 7-Day Peak VRAM |
|---|---|---|
| Whisper ASR (large-v3) | ~7.7 GB | ~10.3 GB |
| TTS voice model | ~4.7 GB | ~4.8 GB |
| nomic-embed-text | ~4.5 GB | ~4.6 GB |
| Total (measured, 7 days) | ~17.0 GB | ~22.8 GB |
At the time of writing, the live reading was 17,775MB used out of 24,576MB — exactly 72%. The three tenants together account for 17.5GB, with the remainder being CUDA context overhead from the GPU itself.
The 23.4GB peak is the genuinely interesting number. It occurred when the Whisper service received a batch of long audio files at the same moment the TTS model was rendering a long response — two large tenants spiking simultaneously while the embedding model held steady at 4.6GB, pushing the GPU within 1.2GB of its ceiling. But it held: no spill, no OOM, no crash — because the peak lasted only a few seconds, not as a sustained load.
The Gap Between Spec-Sheet VRAM and Real-World VRAM
A model's "spec-sheet VRAM" is the weight size. Real-world VRAM is weights + working set + framework overhead. Looking at this gap per tenant produces some counterintuitive conclusions.
Whisper large-v3: FP16 weights are approximately 3.1GB; measured average is 7.7GB — 2.5× the weights. The difference comes from the encoder's audio feature buffers (longer files and queued files consume more), the VAD and alignment models WhisperX runs in parallel, and CUDA context. The 10.3GB peak is a batch of long files all in-flight at once.
TTS voice model: Weights around 4.5GB, measured at 4.7–4.8GB. This is the most "honest" tenant on the card, because TTS is stateless between requests — no long context, no queue buffers.
nomic-embed-text: A 137M-parameter embedding model with spec-sheet weights of roughly 0.3GB, yet measured at 4.5–4.7GB in practice. This overhead comes from the Ollama runtime, CUDA context, and the KV space it pre-allocates for embedding batches. A model that "only needs 300MB" actually consumes 4.6GB as an always-resident service — this is exactly the number that never gets reported.
The pattern is clear: real-world VRAM for always-resident services runs 1.2–2.5× the weight size, with higher multipliers for more stateful models. When sizing a GPU, budget based on measured numbers, not weights.
KV cache and working set are the key to understanding this multiplier gap. Inference frameworks allocate Key-Value caches on the GPU for each active request to store intermediate attention results. For a model like Whisper that processes variable-length audio, this cache grows linearly with audio duration when multiple files are undergoing feature extraction simultaneously. CUDA context itself consumes hundreds of MB, and each independent process (such as Ollama's runtime daemon) initializes its own context — which is why nomic-embed-text's 0.3GB of weights balloon to 4.6GB: Ollama's full runtime, CUDA context initialization, and input/output buffers pre-allocated for batched embedding requests all stack up to produce the real cost. Framework overhead varies significantly by inference engine: vLLM, Ollama, and WhisperX each have their own memory allocation strategies, and some aggressively pre-claim VRAM at startup to reduce inference latency — behavior that is never noted in any spec sheet.
Who Is Actually Consuming the GPU
Seven days of attribution data reveals an interesting distinction: the Whisper tenant consumes the most GPU time, even though the TTS model is the primary contributor to peak usage.
| Caller | GPU-seconds over 7 days |
|---|---|
| Home assistant core (via whisper proxy) | 4,970 |
| Whisper proxy (shared endpoint) | 920 |
| Langfuse worker (observability stack, transcription with audio) | 910 |
| Langfuse MinIO (object storage, transcription at ingest) | 860 |
| Whisper warmup (keep-alive task) | 360 |
| GitHub runner (CI task transcribing test fixtures) | 360 |
This card is a shared service, not a single-application GPU. Over the course of a week, six different callers hit the Whisper endpoint, while the TTS and embedding tenants each had only one caller (the assistant stack). This kind of attribution transforms "the GPU is busy" into "these are the things making it busy" — the difference between a dashboard you glance at and one you can actually act on.
The attribution table above depends on a GPU observability toolchain. nvidia-smi provides instantaneous card-level readings with no ability to distinguish which process or request is consuming VRAM. Achieving per-tenant attribution typically requires instrumentation at each service layer — logging start timestamps, end timestamps, and caller identity for each inference call, then cross-referencing with nvidia-smi dmon or NVML (NVIDIA Management Library) samples to associate usage with time windows. Langfuse is the observability stack mentioned in this article — an open-source LLM operations platform that tracks latency, input/output token counts, and call chains for every inference request, which is why the table can distinguish between "Langfuse worker" and "Langfuse MinIO" as separate callers. This level of granular attribution is missing from most home AI deployments, yet it's the necessary foundation for determining whether a problem is "the model is too large" or "it's being called too frequently."
The Other Side: Is 72% Too Full?
A fair counterargument is that a card running at 72% utilization with only 1.2GB of headroom will fail the day a larger file comes along. Headroom is insurance, and 12% is thin insurance.
The author acknowledges this, but responds with measured data: in real-world use over this week, that peak was a 3-second event, and the GPU absorbed it. The steady state is approximately 17GB, leaving 7.5GB of headroom — 31% of the card. The 23.4GB peak is "the worst case that actually happened," not "the worst case that can be imagined."
The honest version of the headroom argument is: size your GPU for your actual maximum sustained working set, then add 50% headroom — not for the largest peak you can imagine. If your stack's sustained working set is 17GB, a 24GB card is 1.4× the sustained load — sufficient. But if you add a 7B chat model to the same card (roughly 8GB sustained), the sustained working set becomes 25GB and the 24GB card is dead. The real failure mode isn't a peak — it's adding a new tenant.
If Choosing a GPU Today
The author offers sizing recommendations based on measured data:
- Hub card (ASR + TTS + embeddings): 24GB. Sustained working set 17GB, peak 23.4GB. A 16GB card won't work — the TTS model alone would crowd out the Whisper service. 32GB is roomier, but more expensive and hotter for a workload that already fits.
- Chat card: size it to the chat model. A 27B model at FP8 needs two 24GB cards (or one 48GB card). It does not share a card with the hub.
- Golden rule: one card per class of workload. Always-resident services on one card, large chat models on another. Take each workload's measured working set, add 50% headroom — that's the size you need.
The RTX 3090 at 24GB is still the best value per GB on the used market, which is why it's everywhere in home stacks. But "cheapest per GB" is the argument for a chat machine; for a hub machine, the argument is "fits the measured working set of always-resident services with room for one more tenant" — and for this stack, the answer is 24GB, not 16, and not 48.
Conclusion: Meter Readings Beat Spec Sheets
Three AI services on a single 24GB GPU: 17.8GB sustained, 23.4GB peak, zero spill over seven days. Always-resident services consume 1.2–2.5× their weight size in real VRAM; peaks last seconds, not sustained load. Size your GPU by sustained working set plus 50% headroom, and keep large chat models on their own card.
Spec sheets tell you what a single model needs. Meter readings tell you what your entire stack needs. For anyone agonizing over "how much VRAM should I buy" for local LLM inference, this may be a more reliable reference than any estimation formula out there.
Related articles
Is the Agent Harness Dead? Cursor's Early Engineer Breaks Down the Real Architecture of AI Coding Agents
Cursor early engineer Jediah breaks down the real architecture of the agent harness: why "the harness is dead" is only half right, how the data flywheel lets the harness obsolete itself, and how Grokbot, software factories, and primitive design open new frontiers for agents.
Python Visitor Pattern and Better Alternatives Explained
A deep dive into Python's visitor pattern and four alternatives: method pattern, pattern matching, dictionary mapping, and functools.singledispatch. Choose the best decoupling design by change frequency.
357 Days of Deep Experience: Why Omarchy Made Me Stop Distro-Hopping
A 357-day deep dive into Omarchy: 59-second install, out-of-the-box audio and GPU support, the Quattro QuickShell rewrite migration pains, and why Arch users should try it.