Running Qwen 122B on 8GB VRAM: A Complete Guide to Quantized Local Deployment

An "Impossible" Local Deployment Experiment

Most people assume that running a hundred-billion-parameter model requires a high-end GPU cluster. Yet a Bilibili creator known as "小喜同学" managed to run Qwen 122B on an ordinary consumer PC with just 8GB VRAM + 64GB RAM — and even pushed it through a 256K ultra-long context. While "smooth" would be a stretch, this experiment reveals a practical path for local large model deployment.

The original Qwen 122B model uses FP16 precision and weighs around 240GB — far beyond what a typical machine can load. The creator's approach was to use a community-quantized GGUF version, specifically testing the two most-downloaded uncensored quantized models, each compressed to around 50GB.

GGUF (GPT-Generated Unified Format) is a model storage format developed by the llama.cpp project that replaced the older GGML format as the community standard in 2023. The core idea behind quantization is compressing model weights from high-precision floats (e.g., FP16 at 2 bytes per parameter) down to low-bit integers (e.g., Q4 at 4-bit, just 0.5 bytes per parameter), drastically reducing model size and memory footprint. For Qwen 122B, FP16 requires roughly 240GB of storage, while Q4 quantization brings it down to about 61GB. Mixed quantization schemes like Q4_K_M retain higher precision on critical layers, striking a balance between size and capability. Quantization inevitably introduces some precision loss, but research shows that larger models tolerate quantization better — which is exactly why a 122B model can retain strong reasoning ability even after heavy compression. This "trade precision for size" approach is the key to running massive models on ordinary hardware.

Key Parameter Decisions in llama.cpp Deployment

This deployment used llama.cpp (referred to in the video as "老马CBP") build B9934, which started up without any warnings or errors. llama.cpp is an open-source project initiated by Georgi Gerganov in early 2023, originally aimed at running LLaMA models on a MacBook using pure C/C++. It has since evolved into a general-purpose inference engine supporting dozens of model architectures across CPU, GPU, and Apple Silicon. Its core design philosophy is "efficient inference on any hardware," maximizing CPU-side throughput through BLAS acceleration libraries, SIMD instruction sets like AVX2/NEON, and multi-threaded parallelism. Several parameter choices are worth noting for technical readers.

Why the NGL Parameter Was Removed

After multiple tests, the creator deliberately removed the NGL (--n-gpu-layers) parameter. NGL controls how many of the model's front layers are offloaded to GPU VRAM, with the remaining layers handled by the CPU in system memory. When VRAM is insufficient to hold the entire model, partial layer offloading causes frequent data transfers between CPU and GPU, which actually drags down overall speed. The reasoning is straightforward: loading all weights into GPU memory only makes sense when the model fits entirely within VRAM. A 50GB model far exceeds 8GB of VRAM, so rather than wrestling with partial offloading, it's more sensible to load the entire model into 64GB of system RAM and let the CPU handle inference. It's a rational decision grounded in hardware reality.

Context Length and Acceleration Parameters

The context size was set to 256K — what the creator described as "going all in." 256K tokens is roughly equivalent to 500,000 Chinese characters, enough to fit several complete books. However, ultra-long contexts come at a steep cost: in the Transformer architecture, the attention mechanism's computational complexity and memory usage both scale quadratically with context length (O(n²)), and the KV Cache (key-value cache) for a 256K context can reach tens of gigabytes at full precision. The Qwen series uses GQA (Grouped Query Attention) and positional encoding extrapolation techniques like YaRN to reduce long-context resource consumption, but on an 8GB VRAM + 64GB RAM setup, the KV Cache alone consumes a substantial portion of available memory. Temperature, repetition penalty, and other parameters were customized as needed; --no-mmap and related flags were used for acceleration. This configuration pushed the machine to its absolute limits — RAM usage spiked to 98%. The creator quipped they were "just glad they bought the RAM early." For most users, keeping context length between 32K and 128K in practice usually strikes a better balance between speed and capability.

Real-World Performance: 6 tokens/s

The results need to be viewed objectively: average output speed under this configuration was around 6 tokens per second. This clearly can't support high-frequency interaction. The creator admitted honestly that "after running for a while it becomes impractical — you can barely hold a conversation."

But 6 tokens/s isn't without value. The creator did a cross-platform cost comparison: to run the same 122B model in the Apple ecosystem, the cheapest Mac with 64GB of unified memory costs around ¥20,000 (roughly $2,800 USD), and the speed advantage might not be dramatic — the estimate was "maybe hitting double digits, if you're lucky."

Apple's M-series chips use a Unified Memory Architecture (UMA), where CPU and GPU share the same high-bandwidth memory pool, eliminating the data copy bottleneck between CPU memory and GPU VRAM found on traditional PC platforms. Theoretical bandwidth can exceed 100GB/s, far surpassing the ~50GB/s ceiling of ordinary DDR5 memory. This is why, with the same 64GB of memory, Apple Silicon often has a speed advantage for large model inference — but at the cost of significantly higher hardware prices. In other words, achieving usable hundred-billion-parameter inference at very low hardware cost is itself the value of this experiment.

More Cost-Effective Alternatives for Local Deployment

The creator didn't stop at a "showoff" extreme deployment — they also laid out what they consider the most cost-effective local LLM setup:

• Windows 11 PC + NVIDIA GPU: Runs llama.cpp and a search engine (SearXNG), handling GPU-accelerated inference;
• Mac mini M4: Runs natively on macOS for full-featured components, avoiding the pain of wrestling with things like OpenCode inside WSL (as covered in a previous video).

SearXNG is an open-source metasearch engine that can be self-hosted, aggregating results from multiple search engines without exposing user queries to any third party. In local LLM deployment scenarios, SearXNG typically serves as the information source for a RAG (Retrieval-Augmented Generation) pipeline — when a user asks about real-time information or events after the model's training cutoff, the system automatically calls SearXNG to retrieve web summaries and inject them into the context, compensating for the model's knowledge staleness. Compared to commercial search APIs, self-hosted SearXNG fully aligns with the core principle of local deployment: data never leaves your machine.

The creator recommends pairing this setup with a 5060 Ti 16GB GPU, plus a Mac, for a combined cost of around ¥10,000+ (roughly $1,400+ USD). This configuration can run the Qwen 35B Q4 quantized model with 256K context at 60+ tokens/s. The combination essentially divides labor between two heterogeneous systems: the Mac mini handles memory-intensive inference tasks while the NVIDIA PC handles VRAM-friendly tasks — getting the best of GPU performance, native macOS experience, and the flexibility to switch between uncensored models.

Model Capability and the Significance of "Local + Uncensored"

In terms of model capability, the creator's hands-on conclusion was clear: when testing different 50GB-class quantized models with the same prompts, 122B had the highest first-attempt success rate — the advantage of raw parameter scale remains significant. The deeper value lies in the combination of "uncensored + local." No matter how powerful cloud models become, locally deployed open-source models have an irreplaceable role. The creator closed with a pointed statement: technology for good shouldn't just be a slogan, and the power of technology shouldn't be measured by compute alone. When AI sheds its mystique and enters ordinary homes in open-source form, it's not just a democratization of technology — it's dignity and freedom granted to ordinary people in the digital age.

Conclusion: Running Hundred-Billion-Parameter Models Locally Is No Longer a Pipe Dream

The significance of this experiment isn't whether 6 tokens/s is practical — it's that it proves running a hundred-billion-parameter model locally is now feasible. For users who care about data privacy and self-sovereignty, GGUF quantization technology and pragmatic hardware choices are steadily lowering the barrier to large model deployment. The takeaway for everyday developers: rather than blindly piling on compute, find the right balance among model scale, quantization precision, and context length based on your actual hardware constraints.

Key Takeaways