RTX 4090 Running a 125B Model: Qwen3 Next Achieves 100 T/s Inference

Running Hundred-Billion Parameter Models on Consumer GPUs Is Now Real

A long-standing pain point in LLM inference has been that large-scale models typically require expensive professional-grade hardware — think A100s or H100s — to run smoothly. A recent thread on Hacker News sparked discussion around a notable milestone: developers are now able to run the 125-billion-parameter Qwen3 Next Flash model on a single consumer-grade RTX 4090, achieving generation speeds of around 100 tokens/s.

If accurate, this represents a dramatic lowering of the barrier for local deployment of hundred-billion-parameter models. The RTX 4090 has just 24GB of VRAM, and a dense 125B model at FP16 precision requires roughly 250GB of VRAM for weights alone. Getting it to run on a single consumer card necessarily relies on a combination of architectural design choices and inference optimizations.

How 125B Parameters Fit Into 24GB of VRAM

MoE Architecture Is the Key

The Qwen3 Next series uses a Mixture of Experts (MoE) architecture. Although the model's total parameter count may reach the 125B scale, only a small fraction of those parameters are actually activated during each forward pass (i.e., the activated parameter count is far smaller than the total). This is the core reason why MoE models have been able to achieve massive parameter counts on constrained hardware in recent years — sparse activation leads to dramatic reductions in both compute cost and VRAM usage.

In other words, the model's "knowledge capacity" is determined by total parameters, while the "compute cost" of a single inference pass is determined by activated parameters. MoE decouples these two dimensions.

The MoE architecture was systematically introduced by Google in the 2017 paper Outrageously Large Neural Networks, and has since been widely adopted by mainstream open-source models including Mixtral, DeepSeek^Ad, and Qwen. Its core structure replaces the feed-forward network (FFN) layers in a Transformer with multiple parallel "expert" sub-networks, paired with a lightweight router. For each token passing through the layer, the router dynamically selects which experts to activate — typically Top-2 or Top-8 — while the weights of all other experts play no role in that forward pass. For a 125B total-parameter MoE model that activates only ~20B parameters per pass, the actual compute and memory bandwidth consumption approaches that of a 20B dense model, while the model's "knowledge base" remains at the 125B scale. This "activate on demand" design is the fundamental mechanism enabling MoE to support massive parameter counts in VRAM-constrained environments.

Quantization and Memory Offloading

Quantization is equally essential for fitting such a large model into 24GB of VRAM. By compressing weights to 4-bit or lower precision, the model's memory footprint can be reduced to roughly one-quarter of the original. Combined with strategies for offloading some expert weights to system RAM (CPU offloading) or leveraging fast NVMe storage, the full model can be run on consumer hardware.

A generation speed of 100 T/s under this configuration is quite impressive, indicating that the activated parameter count is relatively small and that the inference engine has been well-optimized for data scheduling between VRAM and system memory.

Quantization is the technique of compressing model weights from high-precision floating-point representations (e.g., FP16, at 2 bytes per parameter) to low-bit integer representations. 4-bit quantization means each parameter occupies just 0.5 bytes — a 75% reduction in VRAM usage compared to FP16. Leading quantization methods today include GPTQ, AWQ, and GGUF (from the llama.cpp ecosystem), all of which use compensation algorithms to preserve model quality as much as possible during compression. CPU offloading serves as a complementary strategy: for MoE models, the weights of non-activated experts can reside in system RAM (typically 64–128GB of DDR5), and are only temporarily loaded into GPU VRAM when selected by the router. This introduces PCIe bus transfer latency, but since MoE activates a limited number of experts per pass, the scheduling overhead remains manageable and does not become the primary bottleneck.

Why Local Large Model Deployment Matters

Running a hundred-billion-parameter model on a single RTX 4090 has real practical value for developers and small teams:

• Data privacy: Sensitive data never needs to be sent to a cloud API — all inference happens locally.
• Cost control: After a one-time GPU purchase, there are no per-token fees. The cost advantage becomes significant for high-frequency, long-term use cases.
• Offline availability: No network dependency, making it suitable for edge deployment or restricted environments.
• Customizability: Local deployment makes it easier to fine-tune, integrate custom toolchains, and run experimental modifications.

These benefits used to be limited to smaller 7B or 13B models. MoE architecture has now pushed the boundary of "running large models locally" into the tens-of-billions and even hundreds-of-billions parameter range.

Measured Voices in the Community Discussion

The thread on Hacker News received 89 upvotes and 32 comments, generating meaningful discussion. The community generally approaches claims about "running large models on consumer hardware" with a mix of excitement and caution.

Common follow-up questions center on a few key areas: Is the 100 T/s figure for single-request generation or batch throughput? What quantization precision was used, and is there any perceptible loss in model quality? What are the actual context length limits and time-to-first-token latency? These details often determine whether a setup is truly production-ready or merely a proof-of-concept demo.

For developers considering this approach, it's worth examining these metrics carefully before deployment and evaluating whether the quality trade-offs from quantization are acceptable for your specific use case.

Conclusion

From requiring professional-grade clusters to running hundred-billion-parameter models on a single consumer GPU — the combination of MoE architecture and quantization is reshaping what's feasible for local large model deployment. Qwen3 Next achieving 100 T/s on an RTX 4090 is a vivid data point in this broader trend. As inference engines and hardware continue to evolve, running high-performance large models locally will no longer be the privilege of a select few.