Running LLMs Locally on 8GB VRAM: A Model Selection Guide for Budget Hardware

The Local LLM Struggle on Budget Hardware

A post on r/LocalLLaMA captured the reality facing countless local AI enthusiasts with self-deprecating honesty: stuck with 8GB VRAM and 16GB RAM, scrolling through the community every day, waiting for a high-quality model that can run smoothly on their machine.

Brief as it was, the post hit a nerve for a massive segment of entry-level local deployment users. 8GB VRAM is standard on many mid-range gaming GPUs (like the RTX 3060 and RTX 4060), and that spec puts you in an awkward middle ground for running large language models locally — capable, but severely limited in your options.

Why 8GB VRAM Is Such a Bottleneck

Large language models are extremely memory-hungry. The more parameters a model has, the more VRAM it needs. A non-quantized 7B-parameter model loaded in FP16 requires roughly 14GB of VRAM — already beyond what an 8GB card can handle.

This is precisely why budget users depend so heavily on quantization. By compressing model weights from 16-bit floats down to 4-bit or lower, VRAM usage drops dramatically. Even so, 8GB VRAM typically only handles quantized models in the 7B–14B range comfortably. Anything larger requires offloading some layers to system RAM, which significantly slows down inference.

The 16GB of system RAM mentioned in the post serves exactly this purpose — when the model doesn't fit in VRAM, the overflow spills into RAM. But the cost is a noticeable drop in generation speed.

The speed penalty from CPU offloading deserves a closer look. When model layers are offloaded to system memory, data must travel repeatedly between the GPU and CPU over the PCIe bus. PCIe 4.0 x16 has a theoretical bandwidth of around 32 GB/s, while GPU VRAM internal bandwidth (e.g., 288 GB/s on the RTX 4060) is nearly 10× faster. This means any inference step involving offloaded layers gets throttled to less than a tenth of VRAM bandwidth. In practice, offloading half the model's layers to RAM often drops generation speed from a dozen tokens per second to single digits — a significant blow to conversation fluency. So 16GB RAM is more of a "workable but not great" safety net than a real performance scaling solution.

Gemma QAT: The Pragmatic Choice Right Now

The original poster named their current favorite: Gemma 26B QAT, backing it up with concrete numbers — a steady 26 tokens/sec generation speed.

QAT here stands for Quantization-Aware Training, a technique that accounts for quantization effects during the training process itself. Compared to post-training quantization, QAT models maintain much better output quality after being compressed to lower precision — a critical advantage for users with constrained hardware.

26 tokens/sec is quite usable for local deployment — roughly matching the pace of fast human reading, meaning conversations flow without obvious stuttering. Hitting that mark on an 8GB + 16GB setup demonstrates that QAT quantization genuinely strikes a solid balance between quality and efficiency.

Understanding the difference between QAT and PTQ (Post-Training Quantization) is key to appreciating why this matters. PTQ directly compresses weights after training — simple to do, but with more quality degradation, especially below 4-bit where model output can deteriorate noticeably. QAT simulates quantization-induced numerical errors during training, letting the model actively learn to "tolerate" precision loss through iterations. The resulting quantized weights deviate far less from the original full-precision model's behavior. For end users, the same Q4 quantization level in a QAT version typically performs more reliably on precision-sensitive tasks like logical reasoning and code generation — at the cost of the model provider needing to invest significantly more compute to produce that version.

The Hope for Qwen4: Why MoE Architecture Matters

The most telling part of the post is the author's wish for future models: "only hope is Qwen4 35B A3B or something similar."

The "35B A3B" notation points to MoE (Mixture of Experts) architecture. The "A3B" means that while the model's total parameter count is large (35B), only about 3B parameters are actually activated during each inference pass. The clever part: the model's overall capability approaches that of a large model, while the per-inference compute and VRAM pressure stays close to that of a small model.

For budget hardware users, MoE is an extremely attractive direction — it could let someone with 8GB VRAM access model capabilities far beyond what the parameter count suggests. The author even speculated about combining this with ngram techniques to push a 35B model's effective performance toward 70B territory, reflecting the community's intense appetite for "doing more with less" breakthroughs.

The basic principle of MoE (Mixture of Experts) architecture is to replace a model's feed-forward network layers with multiple parallel "expert" sub-networks, with a router dynamically selecting a small number of experts to participate in computation for each token during inference. In a 35B A3B model, roughly 35B parameters are distributed across many experts, but only the experts corresponding to ~3B parameters are activated in each forward pass. This yields two direct benefits: only the weights of the active experts need to reside in VRAM (achievable through dynamic loading), and floating-point operations (FLOPs) drop dramatically, making inference speed comparable to a dense model of equivalent active parameter count. MoE isn't a new concept — its large-scale implementation traces back to Google's Switch Transformer (2021) — but in recent years, open-source models like Mixtral and Qwen have brought it into the view of consumer hardware users.

A Realistic Picture of the Local Deployment Ecosystem

This seemingly casual post reflects several realities of the local LLM community:

First, the entry-level hardware user base is large and active. They're not chasing peak performance — they're looking for the best possible solution within a limited budget.

Second, quantization (especially QAT) has become the core enabler for low-spec deployments. Whether a model provider offers a high-quality quantized version directly determines how well-received that model is in the community.

Third, MoE architecture is being pinned with high hopes. As teams like Qwen and Mistral continue releasing MoE models, the approach of "unlocking large model capability with small active parameter counts" may well become the key breakthrough for democratizing local AI.

For anyone considering getting started with local LLMs, this post delivers a clear message: 8GB VRAM isn't a dead end. With the right quantized model and a sensible RAM offloading strategy, a genuinely useful local AI experience is within reach. And as MoE architecture becomes more widespread, the hardware bar for high-quality models is likely to drop even further.