Running an 84GB LLM on a Single 7900XTX: 100+ Token/s Benchmark and Tuning Breakdown

An AMD 7900XTX paired with 64GB of RAM running an ~84GB model at over 100 tokens per second — this result challenges most assumptions about deploying large models on consumer-grade GPUs. Based on a hands-on benchmark video, this article breaks down the real-world performance, model selection trade-offs, and the tuning logic behind this setup.

Benchmark Results: Prefill Exceeds 1000, Decode Holds Steady at 90+

The test used Qwen 3.8 Flash Next (Q3 quantized), with weights totaling approximately 84GB, running on an AMD 7900XTX with 24GB of VRAM. Data from the profile+decode performance testing tool paints an impressive picture.

Prefill throughput increases with prompt length: around 290 token/s for short prompts, climbing to roughly 1300 token/s for longer contexts, with an average prefill throughput of 900 token/s. This means longer inputs actually process faster rather than slower — a genuinely useful property for real-world workloads.

Decode speed remains relatively stable at around 90 token/s, with total output throughput ranging from 63 to 1100, averaging approximately 90 token/s. It's worth noting that tests were capped at a prompt length of 8192 tokens; prefill speed would theoretically increase further with longer contexts.

In a practical generation task, the tester prompted the model to write a rover exploration game playable in a browser. After hitting send, the model paused for only a few seconds before producing a dense stream of tokens, generating approximately 2800 tokens of complete code in a single pass at an average output speed of 72 token/s — code that ran correctly after being copied directly. This was actually faster than a previously tested Qwen 3.8 27B model.

Intelligence Comparison: On Par with DeepSeek Flash, Crushes 27B Q4

Speed alone doesn't determine usability — actual model capability matters just as much. Three models were compared: DeepSeek^Ad Flash as the baseline, Qwen 3.8 27B Q4, and Qwen 3.8 Flash Next Q3KL.

The Q3-quantized Qwen 3.8 Flash Next showed a slight capability drop compared to the Q3KL version, but was nearly on par with DeepSeek Flash. Against Qwen 3.8 27B Q4, however, it showed a clear advantage — as the tester put it, the 27B Q4 is "a whole tier below" the Flash version in terms of intelligence.

This comparison offers a clear model selection heuristic: if your hardware can run the Flash version, choose it over the smaller 27B Q4. A larger base model, even with aggressive quantization, generally outperforms a lightly quantized smaller model.

---

What is Q3 quantization? Q3 quantization compresses model weights from the original 16-bit or 32-bit floating point representation down to approximately 3 bits per parameter, reducing storage to roughly one-fifth of the original size. Lower quantization precision means a coarser range of representable values, which theoretically introduces some capability loss — though the actual impact varies depending on model architecture and quantization algorithm. Q3KL is a variant based on KL divergence (Kullback-Leibler divergence) minimization, which preserves the shape of the original weight distribution during compression. It typically performs more consistently on intelligence benchmarks than naive Q3 but is slightly slower. The article notes that Q3 shows a minor intelligence dip versus Q3KL, but the further reduced weight size makes the entire setup viable within the limited VRAM + RAM constraints — a deliberate trade-off between deployment feasibility and model capability.

---

How 84GB of Weights Fit Inside a 24GB GPU

This is the most counterintuitive part of the setup. With only 24GB of VRAM and 84GB of model weights, the solution is a hybrid loading strategy that spans both GPU and system RAM.

The inference engine used is SGLang (referred to verbally in the video as the "Straight engine"), combined with an MTP (Multi-Token Prediction) draft mechanism to accelerate decoding. The deployment process is comprehensive: customizing and compiling the inference framework, installing base dependencies, cloning the source code, targeted build configuration, initializing a Python virtual environment, downloading model weights, preparing MTP drafts, and deep modifications to the engine itself.

One important clarification: while the video title says "single GPU," the tester acknowledges that the actual test environment used two GPUs running on PCIe x8 bandwidth. He speculates that removing the second card and running a single GPU at full PCIe x16 speed could improve performance further. This is a critical detail for anyone looking to replicate this setup — actual single-GPU performance needs to be independently verified.

---

How does the GPU + RAM hybrid loading strategy work? This technique is technically known as "CPU offloading" or "unified memory inference." The inference framework distributes model weights layer by layer: compute-intensive layers are kept in VRAM to leverage GPU parallelism, while the remaining layers reside in system RAM (main memory) and are swapped into VRAM on demand during each forward pass, then swapped back out. The bottleneck is PCIe bus bandwidth — the speed at which weights move between RAM and VRAM directly limits decode throughput. In this test, the PCIe x8 configuration provides roughly 16 GB/s of theoretical bandwidth, which is why the tester expects a further speed boost when switching to single-card PCIe x16. The 64GB RAM capacity here serves not only as a size guarantee ("will it fit?") but also affects the memory subsystem's total bandwidth, meaning RAM frequency and channel count have a tangible impact on overall performance.

---

Tuning Is What Makes the Performance

The tester emphasized that initial speeds were far lower — it was a series of optimizations that pushed the result past 100 token/s.

Key tuning actions included: upgrading the decode window from 2 to 4; eliminating breakpoint appends in speculative sampling itself; and optimizing MTP long-context handling — the original 256-token setting was optimized to 128K. The tester's view is that 128K context is more than sufficient for typical tasks, and for extremely long inputs, a context compression pass is enough — the speed gains from this change are substantial.

These details reveal that the performance ceiling for consumer hardware running large models depends heavily on fine-grained framework tuning, not just raw hardware. With the same GPU and RAM, there can be an order-of-magnitude difference in speed between default settings and a deeply tuned configuration.

---

What is the MTP draft mechanism? MTP (Multi-Token Prediction) is a variant of Speculative Decoding. The standard speculative decoding approach uses a small "draft model" to predict multiple candidate tokens at once; the main model then verifies them in parallel, accepting correct predictions and discarding incorrect ones, effectively advancing multiple tokens per forward pass and boosting decode speed. MTP embeds the draft prediction capability directly into additional output heads within the main model, eliminating the overhead of maintaining a separate draft model. Upgrading the decode window from 2 to 4 means the draft predicts 4 candidate tokens per round — the higher the verification acceptance rate, the more speed is gained. This class of optimization essentially breaks through the per-token-generation speed ceiling of autoregressive decoding through batch speculation and parallel verification, without meaningfully compromising model intelligence.

---

Implications for Local Deployment

This benchmark provides a valuable reference point: with one (or two) consumer-grade AMD GPUs and 64GB of RAM, it's possible to run a local model with intelligence close to DeepSeek Flash at over 100 token/s output speed, capable of handling real-world tasks like code generation reliably.

For individual developers and small teams who care about data privacy, want local deployment, and aren't willing to invest in server-grade hardware, this SGLang + MTP + quantization combination offers a viable path. The tester also provides complete documentation covering hardware selection, PCIe topology, system environment setup, and launch scripts — and mentions the documentation can be handed directly to an agent tool for one-click deployment.

A few caveats: the "single GPU" framing in the video title doesn't match the actual dual-GPU test environment, and Q3 quantization does involve real precision trade-offs. Anyone looking to replicate this setup should follow the tester's detailed documentation carefully and run their own benchmarks against their specific hardware.