Running Qwen3.8-Flash-Next on a 12GB GPU: Custom Engine Hits 65 tok/s

A developer shared an impressive local inference optimization on Reddit: running the Qwen3.8-Flash-Next model on an RTX 5070 with just 12GB of VRAM, achieving approximately 65 tokens/s output speed and a prompt processing speed of 543 tokens/s. These numbers are quite remarkable for consumer-grade hardware, and once again demonstrate that local LLM deployment is rapidly approaching practical viability.

From 15 to 65 tok/s: A Major Performance Leap

The author noted that previously, when running an IQ3_XXS quantized model with llama.cpp, output speed was only 15 tok/s with prompt processing at 100–120 tok/s. After building a custom inference engine specifically for this model and PC configuration, the same IQ3_XXS quantization achieved roughly 65 tok/s output and 430 tok/s prompt processing — nearly a 4× improvement.

The key to this leap lies in targeted optimization. General-purpose inference frameworks must accommodate a wide variety of model and hardware combinations, while an engine built for a single model and a single hardware tier can squeeze every last drop out of scheduling, memory layout, and compute paths. The author named it the Strata engine and has released it as open source.

llama.cpp is currently the most mainstream open-source local LLM inference framework, developed by Georgi Gerganov in 2023. Its core strength lies in broad cross-platform and cross-hardware compatibility, supporting CPU, CUDA, Metal (Apple Silicon), and more. Precisely because it must support so many hardware and model combinations, llama.cpp has to make numerous generality compromises in memory layout, scheduling strategy, and compute kernels — it simply cannot be fine-tuned to the extreme for any specific model or GPU architecture. By contrast, the Strata engine sacrifices generality in favor of deep adaptation to Qwen3.8-Flash-Next's model architecture (such as attention head count and inter-layer dependencies) and the RTX 5070's hardware characteristics (such as Tensor Core utilization on the Ada/Blackwell architecture and L2 cache size). This allows it to extract nearly 4× throughput improvement at the same quantization precision. The performance gap between "specialized engine vs. general framework" has precedent in industrial inference scenarios; seeing it replicated on consumer hardware here is equally noteworthy.

Hardware Configuration and Quantization Approach

The author's test platform is not a high-end build — in fact, it's quite representative of mid-range consumer setups:

• GPU: 12GB RTX 5070 SFF (Gigabyte)
• RAM: 64GB DDR5-5600
• CPU: Ryzen 5 7600
• OS: Windows

The total cost of this setup falls well within the consumer range, yet it can run a quantized model that requires nearly 40GB of combined memory and VRAM. The core idea is RAM–VRAM co-loading: the portion that doesn't fit in VRAM is offloaded to the large system memory, with 64GB DDR5 carrying the full model.

Also worth noting is the RCO-GSQ quantization technique the author uses. He mentioned that 2-bit quantization runs faster under this scheme, with the model hosted on Hugging Face (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF). This quantization method is associated with ISTA-DASLab's research and points toward a more cutting-edge low-bit quantization approach.

RCO-GSQ is a low-bit quantization scheme for large models proposed by ISTA-DASLab (the Institute of Science and Technology Austria's Data Analytics and Learning Lab). GSQ (Grouped Sparse Quantization) reduces precision loss by grouping weights and introducing sparsity, while RCO (Residual Compression and Optimization) further compresses and corrects quantization residuals on top of that — enabling relatively acceptable model quality at extremely low bit rates such as 2-bit. Traditional 2-bit quantization is often unusable due to excessive precision loss; RCO-GSQ improves on this through finer-grained error compensation. This is also why the author noted that "2-bit runs faster under this scheme" — at an acceptable precision level, lower bit counts mean less memory bandwidth pressure and faster matrix operations, and RCO-GSQ's technical guarantees allow this precision-speed trade-off point to shift forward.

Detailed Performance Numbers

At a 128K context length, three quantization tiers performed as follows:

Output Speed (tokens/s)

Quantization	Effective Quality	Output Speed
Q2_0	Equivalent to unsloth Q3	65.1
IQ2_XS	Equivalent to unsloth Q4	52.0
IQ3_XXS	Equivalent to unsloth Q5	44.8

Prompt Processing Speed (tokens/s)

Quantization	Prompt Processing
Q2_0	543
IQ2_XS	472
IQ3_XXS	414

Memory Requirements (RAM + VRAM combined)

• Q2_0: minimum 37.6GB
• IQ2_XS: minimum 39.2GB
• IQ3_XXS: minimum 47GB
• Vision encoder adds an additional 0.91GB

The data clearly illustrates the trade-offs between quantization tier, speed, quality, and memory footprint: more aggressive quantization (Q2_0) yields higher speeds and lower memory usage but at relatively lower quality; IQ3_XXS offers higher precision at the cost of a near-47GB memory requirement and slower inference.

The "effective quality" comparisons in the table deserve extra clarification. The IQ series (e.g., IQ2_XS, IQ3_XXS) is the Importance Quantization format introduced by llama.cpp. Rather than applying a uniform bit width to all weights, it allocates different levels of precision based on each weight's importance to model output — critical weights retain higher precision while less important weights are compressed more aggressively. This means that although IQ2_XS is nominally a 2-bit quantization, its actual average bit rate is slightly higher, and its model quality is comparable to uniform 4-bit quantization (i.e., unsloth Q4). Understanding this is crucial for interpreting the table: Q2_0's "65.1 tok/s" does not mean it uses the same amount of information as IQ2_XS — Q2_0 is uniform 2-bit, lower quality but fastest; IQ2_XS trades a small amount of speed for effective Q4-level precision, offering a better Pareto point between speed and quality.

An Open-Source, One-Click Solution

For everyday users, the most valuable aspect of this work is its accessibility. The author says the engine now supports one-click installation and currently targets CUDA only. All relevant resources are publicly available:

• Engine code: The Strata project on GitHub
• Quantized model: The GSQ-RCO-GGUF version on Hugging Face

This means users with similar setups can reproduce this performance directly, without needing to wade through complex compilation and tuning processes.

It's worth noting that this solution is highly optimized for a specific model and hardware configuration, and has limited generality. It's more of a showcase of "how to push consumer hardware inference performance to the limit" rather than a universal framework that can be freely ported to other models. The current CUDA-only support also means AMD or Apple Silicon users cannot benefit for now.

The Significance of Local Inference

The value of this kind of work lies in how it continually lowers the hardware barrier to running large models. When a mid-range GPU with 12GB of VRAM, paired with large-capacity system memory, can smoothly run Qwen3.8-Flash-Next with vision support at 65 tok/s, local deployment becomes increasingly realistic for users who prioritize privacy, need offline capability, or want to avoid API costs.

The case of a custom engine delivering nearly 4× speedups also points to a broader direction: beyond general-purpose frameworks, there remains enormous performance headroom to be unlocked through deep optimization targeting specific models and hardware.