Unsloth Releases Qwen3.6 NVFP4 Quantization: 2.5x Faster with Zero Accuracy Loss

Another Breakthrough in Quantized Inference

In the field of local large model deployment, quantization has always been a core technique for balancing performance and resource consumption. Quantization refers to the technique of compressing floating-point parameters in a neural network (typically 32-bit or 16-bit) into lower-precision representations—it not only reduces the model's storage footprint but, more critically, allows GPU hardware to process more computational units per unit of time, thereby boosting inference throughput. From INT8 and INT4 to FP8 and FP4, the evolution of quantization precision has always been accompanied by a trade-off between accuracy loss and speed gains.

The evolution of quantization technology has a deep engineering background. Early research found that a large number of parameters in neural networks are far less sensitive to numerical precision than expected during the training phase, providing the theoretical foundation for low-precision inference. From PyTorch's dynamic quantization to TensorRT's static quantization, and then to post-training quantization algorithms designed specifically for large language models such as GPTQ and AWQ, this field has undergone explosive development over the past three years. The evolutionary path of these algorithms clearly reflects the core contradiction of large language model inference engineering: the ongoing tension between the exponential growth of model parameter counts and the linear growth of consumer-grade hardware VRAM capacity. GPTQ approximates layer-by-layer optimal quantization through second-order Taylor expansion, significantly reducing accuracy loss; AWQ, through activation-aware weight quantization—protecting the weight channels sensitive to activation values rather than quantizing uniformly—achieves near-full-precision performance at INT4 precision without requiring a dedicated calibration dataset. NVFP4 represents the latest stage of this evolution—extending the quantization format from the integer domain to the floating-point domain in exchange for greater dynamic range and better numerical stability, marking a fundamental shift in the precision representation paradigm.

The renowned open-source team Unsloth released an NVFP4 quantized version of the Qwen3.6 series models in the r/LocalLLaMA community, claiming up to 2.5x inference acceleration compared to NVIDIA's official NVFP4 quantization implementation, with almost no accuracy loss across multiple benchmarks.

This achievement not only pushes the performance ceiling of the NVFP4 format but also provides users with hardware supporting FP4 Tensor Cores a more efficient option for local inference.

reddit source: 2.5x faster Qwen3.6 NVFP4 Unsloth quants

Technical Core: W4A4 vs W4A16, What's the Difference?

True 4-bit Tensor Core Computation

The key to Unsloth's speedup lies in the choice of computation precision. NVIDIA's official NVFP4 quantization uses a W4A16 scheme—weights are 4-bit, but activations remain 16-bit. There's an easily overlooked detail here: in W4A16, although weights are stored in 4-bit, they must first be dequantized back to high precision before being multiplied with 16-bit activations. The actual matrix multiplication is still performed at FP16 precision, rendering the 4-bit Tensor Cores useless and leaving the hardware's potential far from fully utilized.

Unsloth's approach adopts W4A4: both weights and activations are 4-bit, and matrix multiplication is executed directly on the hardware's 4-bit Tensor Cores, theoretically achieving throughput several times that of W4A16. This difference is the fundamental reason the 27B model achieves 2.5x acceleration.

To understand why W4A4 can deliver several-fold acceleration, one needs to understand the working mechanism of modern GPU Tensor Cores. NVIDIA Tensor Cores are hardware units designed specifically to accelerate matrix multiplication, and their throughput is directly tied to operand precision: FP16 Tensor Core compute is typically 8x that of FP32, while FP4 Tensor Cores can theoretically reach 4x that of FP16. The key constraint is that Tensor Cores require the two operands of matrix multiplication to have exactly matching precision to trigger the hardware acceleration path—this hardware design determines the fundamental limitation of the W4A16 scheme. In the W4A16 scheme, after weights are dequantized back to FP16, the actual multiplication is completed at FP16 precision, and the 4-bit Tensor Cores are completely unused; whereas the W4A4 scheme keeps both weights and activations at 4-bit, directly taking the FP4 Tensor Core hardware acceleration path, fundamentally unleashing the peak compute of the Blackwell architecture. The difficulty of activation quantization lies in the dynamic nature of its distribution, which cannot be calibrated offline like weights. Unsloth's precise calibration at this stage is the core engineering challenge in implementing the W4A4 scheme while maintaining accuracy.

It's worth mentioning that the NVFP4 used here is a 4-bit floating-point format designed specifically by NVIDIA for the Blackwell architecture (GB200, RTX 50 series), employing E1M2 encoding with 1 sign bit, 1 exponent bit, and 2 mantissa bits. Compared to INT4 integer quantization, NVFP4 has a larger dynamic range and can better represent the unevenly distributed weights and activations in neural networks. The theoretical compute of NVFP4 Tensor Cores on the Blackwell architecture is about 4x that of FP16, making it a core hardware feature of NVIDIA's next-generation inference acceleration.

Layered Design of Precision and Speed

For the 35B-A3B model, Unsloth provides two versions to meet different scenario requirements:

NVFP4 (1.56x speedup): Uses a mixed-precision strategy, retaining higher precision for certain layers to prioritize accuracy;
NVFP4-Fast (1.79x speedup): Fully adopts W4A4 in pursuit of ultimate inference speed.

The "A3B" designation in Qwen3.6-35B-A3B reveals the model's architectural characteristics: this is a Mixture of Experts (MoE) model with a total parameter count of about 35B, but only about 3B parameters are activated per inference. The MoE architecture uses a routing mechanism (Router) to distribute different inputs to different expert sub-networks, decoupling parameter count from compute cost—the model gains the knowledge capacity that a large parameter count provides while keeping inference cost close to a 3B dense model. This architectural characteristic presents unique challenges for MoE model quantization: the sparsity of expert routing leads to more irregular activation distributions, and the significant differences in weight distributions across different expert sub-networks impose far higher calibration requirements than dense models—some experts may rarely be activated throughout the entire inference process, and their weight quantization errors are difficult to fully cover with a conventional calibration dataset. Unsloth provides both standard and Fast configurations for this reason, which is essentially a mixed-quantization strategy that retains higher precision in routing-sensitive layers. This layered design allows users to freely make trade-offs based on actual needs, rather than being forced to accept a single compromise solution.

Performance Data: Overview of Acceleration Across Models

According to data released by Unsloth, the inference acceleration performance of each version is as follows:

Model Version	Speedup vs NVIDIA NVFP4
Qwen3.6-27B-NVFP4	2.5x
Qwen3.6-35B-A3B-NVFP4	1.56x
Qwen3.6-35B-A3B-NVFP4-Fast	1.79x

Beyond inference speed, this quantized version also has built-in FP8 KV Cache calibration. The KV Cache is a VRAM region in Transformer inference used to cache the Key and Value matrices of the attention mechanism. Its size scales linearly with sequence length and is the main VRAM bottleneck for long-context inference.

FP8 KV Cache compression is not a simple precision truncation but requires precise dynamic range calibration: the numerical distribution of Key and Value matrices in the attention mechanism varies by layer, requiring a separate quantization scaling factor to be determined for each layer. An uncalibrated FP8 KV Cache may cause attention score distortion, subsequently leading to "forgetting" or "hallucination" problems in long-text inference. Unsloth's built-in FP8 KV Cache calibration statistically analyzes the activation distributions of each layer on a representative dataset, pre-determining optimal scaling parameters to ensure precision compression stays within an acceptable range for model quality. Notably, there is a fundamental methodological difference between KV Cache quantization calibration and weight quantization calibration: weights are static and can be calibrated once offline; whereas the numerical distribution of the KV Cache depends on runtime input content and requires sufficiently diverse calibration samples to statistically cover the distribution range of actual deployment scenarios, which is why this stage has higher engineering complexity. Compressing the KV Cache from BF16 (16-bit) to FP8 (8-bit) can halve its VRAM footprint, theoretically supporting double the context length—this is exactly the technical basis for the version's claim of "automatic support for 2x longer context," effectively alleviating VRAM pressure in long-document and long-conversation scenarios.

The model also has MTP (Multi-Token Prediction) pre-embedded. MTP allows the model to predict multiple subsequent tokens simultaneously in a single forward pass rather than generating them one by one. Combined with the Speculative Decoding mechanism, it can significantly boost generation throughput, with particularly notable effects in batch inference scenarios.

The combination of MTP and speculative decoding represents the current mainstream paradigm for improving inference throughput. In traditional autoregressive generation, each token's generation requires a full forward pass, and GPU utilization during the decoding phase is typically below 20%—much compute is spent waiting for the previous token to finish generating rather than on actual computation. Speculative decoding introduces a lightweight draft model to pre-generate multiple candidate tokens, which are then verified in parallel by the main model, compressing multi-step serial decoding into a single parallel verification, theoretically achieving 2-4x throughput improvement without loss of output quality. MTP internalizes this idea into the model itself: by adding multi-token prediction heads during the training phase, the main model gains the ability to directly predict multiple future tokens without an additional draft model, achieving similar acceleration while keeping the architecture simple. The advantage of this approach is that it eliminates the distribution mismatch problem between the draft model and the main model, while also avoiding the deployment complexity of maintaining two independent models, making it especially suitable for local single-user deployment scenarios.

Accuracy Validation: Comprehensive Comparison Across Multiple Benchmarks

Speed improvement is meaningless if it comes at the cost of model quality. Unsloth conducted a comprehensive horizontal comparison of FP8, BF16, NVIDIA NVFP4, and its own NVFP4 across three authoritative benchmarks: MMLU-Pro, AIME 2025, and GPQA.

The combination of these three benchmarks covers reasoning abilities across different dimensions: MMLU-Pro is an upgraded multi-discipline academic understanding test, covering 12,000 questions across 14 disciplines including mathematics, physics, and law, and is more challenging than the original MMLU—its questions deliberately add distractor options and raise the depth requirements of reasoning; GPQA (Graduate-Level Google-Proof Q&A) focuses on doctoral-level biology, chemistry, and physics problems, measuring the model's deep scientific reasoning ability—the questions are deliberately designed to ensure that answers cannot be obtained through simple web searches and must rely on genuine domain expertise. Statistics show that even human experts holding doctoral degrees in the relevant field achieve an average accuracy of only about 65% on this test; AIME 2025 comes from the American Invitational Mathematics Examination and is a standard test set for evaluating a model's competition-level mathematical problem-solving ability. Its questions require multi-step precise reasoning and place extremely high demands on the completeness of the logical chain. Together, the three constitute an authoritative benchmark combination for evaluating the comprehensive reasoning level of large language models today, stress-testing quantized models comprehensively across three dimensions: breadth of knowledge understanding, depth of professional reasoning, and precise mathematical derivation.

Qwen3.6-27B Test Results

Provider	MMLU-Pro	GPQA	AIME 2025
Unsloth	86.25	86.34	93.12
NVIDIA	85.96	86.87	93.12
FP8	86.11	86.87	93.75
BF16	85.96	88.13	93.33

Unsloth's NVFP4 quantization scored 86.25 on MMLU-Pro, slightly surpassing the BF16 full-precision version's 85.96, with the remaining metrics also roughly on par with the official version, proving that its quantization strategy effectively retains the original capabilities while compressing the model.

Qwen3.6-35B-A3B Test Results

Provider	MMLU-Pro	GPQA	AIME 2025
Unsloth	85.85	86.74	92.29
Unsloth Fast	85.58	87.75	91.67
NVIDIA	85.60	87.12	91.88
FP8	85.75	86.74	93.12
BF16	85.75	86.36	92.50

On the 35B model, Unsloth's standard version's MMLU-Pro (85.85) again leads the NVIDIA version (85.60) and BF16 (85.75). Even the ultra-fast Fast version keeps the accuracy drop within a very small range—an AIME 2025 score of 91.67, less than 1.5 percentage points behind the other versions.

Significance for the Local Deployment Ecosystem

Lowering the Barrier to Running High-Quality Large Models

Unsloth was founded in 2023 by brothers Daniel Han and Michael Han, initially gaining fame for dramatically accelerating fine-tuning of the LLaMA series models—its custom CUDA kernel implementation boosted LLaMA-3 fine-tuning speed to more than 2x that of mainstream frameworks on the Alpaca dataset while reducing VRAM usage by about 60%. The team's core technical approach is to bypass high-level framework abstractions and write fused kernels (Fused Kernels) directly targeting specific hardware architectures, eliminating intermediate data movement overhead—this methodology, known as "Kernel Fusion" in the field of deep learning optimization, essentially merges multiple serial GPU operations into a single GPU call, thereby reducing VRAM read/write counts and scheduling overhead, with particularly significant effects in bandwidth-limited inference scenarios. In the quantization domain, Unsloth continues this methodology: rather than relying on the generic implementations of bitsandbytes or TensorRT-LLM, it customizes the computation graph for target quantization formats and hardware characteristics, which is the fundamental reason it can surpass NVIDIA's official implementation.

This NVFP4 quantization of Qwen3.6 further lowers the hardware barrier to running high-quality large models. For users with GPUs supporting FP4 Tensor Cores (such as the Blackwell architecture), 2.5x acceleration means lower inference costs and faster response times.

NVFP4 Format Reaches Maturity

As NVIDIA's flagship next-generation 4-bit floating-point format, NVFP4 has an advantage in dynamic range over traditional integer quantization (such as INT4), enabling more precise representation of the diverse numerical distributions of neural network weights and activations. Unsloth's practice proves that through careful W4A4 design and calibration, NVFP4 can fully unleash hardware potential while maintaining accuracy, offering positive demonstrative value for the evolution of the entire quantization ecosystem.

For developers who want to try it hands-on, Unsloth has provided complete model links and detailed benchmark analyses in its official documentation. As hardware supporting FP4 computation continues to become more widespread, such quantization solutions are poised to become one of the mainstream choices for high-performance local inference.

Note: All data in this article comes from the Unsloth team's official release on r/LocalLLaMA. Actual performance may vary depending on hardware and inference framework.