Unsloth Releases Quantized DeepSeek-V4.1 GGUF: Low-Bit Precision Beats ISTA

Can quantized models preserve accuracy while shrinking in size? That has always been the most practical tension when deploying large models locally. Unsloth's latest answer is rather provocative: their DeepSeek^Ad-V4.1-Flash-1017 Dynamic V3.1 (preview) GGUF not only outperforms the previous v3 generation on multiple benchmarks, but also beats ISTA's GSQ-RCO quantized versions across nearly the board.

According to Unsloth's official Reddit release post, the team reproduced and tested ISTA's GSQ-RCO GGUFs but could not obtain the results ISTA reported—even when using the highest scores among them. The most contentious item is AIME25: ISTA claims 100% accuracy recovery over the baseline, while Unsloth's own measurements show only about 91.88%.

Benchmarks: Who's Swimming Naked Is Obvious

Unsloth's comparison covers AIME25, HMMT Feb26, GPQA-D, MMLU-Pro, IFEval, LiveCodeBench v6 (32K), and Aider pass@1. To ensure credibility, each benchmark was run at least 3 times, and AIME25 was run 16 times with the average taken.

Looking at the published data tables, Unsloth's UD series leads on almost all metrics. Even their self-described "old" v3 outperforms ISTA's corresponding versions on the vast majority of metrics. One notable detail is the size-versus-precision trade-off: UD-IQ1_S is only 233.0GB yet scores 92.92 on AIME25, while the larger (265.8GB) GSQ-RCO IQ2_XS scores a comparable 95.00 on AIME25—but on HMMT Feb26, UD-IQ1_S scores 57.01 versus GSQ-RCO IQ2_XS at 64.39. This shows that different quantization schemes have inconsistent strengths across benchmarks, and you cannot draw conclusions from a single metric alone.

It should be noted that these numbers come from Unsloth's own testing environment and procedures—vendor-side self-reported results. Independent third-party reproduction remains a key part of judging quantization schemes. The original post also explicitly states that complete benchmark data, methodology, and charts are available on their blog.

GGUF is the single-file model format used in the llama.cpp ecosystem, packaging weights, tokenizer, and metadata together, and supporting multiple quantization types (such as Q4_K_M, IQ2_XS). Those with the IQ prefix belong to the i-quant series, which uses finer codebooks and importance matrices to reduce bit count; UD is Unsloth Dynamic's custom quantization scheme, which claims to dynamically allocate bits across layers. Therefore, although UD-IQ1_S and GSQ-RCO IQ2_XS in the comparison tables are both based on GGUF, their quantization strategies and average bit counts differ—which is exactly where the size differences come from.

Can KLD Really Measure Quantization Quality?

There is another technical discussion point in the release post: is KLD (KL divergence) reliable? Unsloth's explanation is straightforward—the primary goal of quantization is to minimize KLD while minimizing disk footprint. If KLD is 0, it means the quantized model is equivalent to the 16-bit model, with logits perfectly modeled.

According to their tests, UDv3.1 is clearly better than GSQ-RCO on KLD and top-1% metrics across multiple domains including coding, agentic coding, and conversation. This actually addresses a common misconception: many people focus only on "how much the benchmark score drops after quantization" while ignoring distribution-level shifts. Low KLD does not guarantee high task scores, but it reflects whether the model's output distribution has been damaged—the two should be considered together.

Another issue they proactively acknowledge: low-bit quantized models do indeed "think longer" during inference than unquantized models. Unsloth says they plan to further investigate and optimize quantization quality, but they still lead GSQ-RCO on inference length, and the blog provides relevant charts on truncated responses.

KL divergence (KLD) measures the difference between two probability distributions: let P be the output distribution of the unquantized 16-bit model and Q be that of the quantized model; KLD is the average information loss of P relative to Q. A KLD of 0 means Q is perfectly consistent with P, and quantization has not damaged any logits information. In practice, KLD is typically accumulated token by token over the entire calibration corpus and then averaged, with metrics such as top-1% characterizing the worst case.

It is important to note that KLD is a distribution-level average and does not strictly monotonically correspond to downstream task accuracy: a model with slightly higher KLD may happen to score better on certain types of tasks. Therefore, KLD is better suited as a "health check metric" for quantization quality, used to determine whether the model has been systematically damaged, while task scores reflect specific capabilities. The two are complementary rather than substitutes.

Hardware Requirements and New Desktop Features

In terms of deployment requirements, this model can run on 128GB RAM or unified memory devices, and with engram offloading it can even run on 64GB memory devices. For developers who want to run large models locally, this is a relatively friendly signal—especially for devices with unified memory architecture.

Also released in the same batch are a set of new capabilities for Unsloth Desktop: Browser Use, auto compaction, a new TTS pipeline (supporting voice cloning, separation, etc.), sandboxing, and background customization. It is clear that Unsloth is not just producing quantized weights but also building out a complete local inference workflow.

engram offloading refers to keeping relatively static, predictable parts of the model (such as some expert weights or embedding tables) resident in memory/SSD and swapping them into the compute unit when needed, thereby breaking through the hard limits of VRAM or unified memory capacity. This differs from traditional KV cache offloading, which offloads the activation cache that grows dynamically during inference, whereas engram offloading offloads weight-side content. It is therefore more effective for scenarios like 64GB devices where "memory is not enough to hold the entire model," at the cost of additional data transfer latency.

Take Vendor Self-Tests with a Grain of Salt

Overall, there are two core messages from this release: first, Unsloth Dynamic V3.1 holds an advantage over GSQ-RCO on multiple benchmarks, and low-bit versions approach or even exceed higher-bit older schemes on some metrics; second, the discussion around KLD brings quantization evaluation back from "benchmark scores" to the more fundamental level of "distribution fidelity."

But conclusions still require caution. All comparison data is produced by Unsloth, and ISTA has not yet responded in that thread. For ordinary users, the more pragmatic approach is to run small-scale validation on your own tasks rather than directly copying leaderboard results. After all, the choice of quantization scheme ultimately depends on your hardware, task type, and tolerance for inference latency.

One-line Summary

Unsloth releases quantized DeepSeek-V4.1 GGUF, self-tests show it beats ISTA on multiple benchmarks.

Paragraph Summary

Unsloth releases DeepSeek-V4.1-Flash-1017 Dynamic V3.1 (preview) GGUF, claiming near-sweeping leadership over ISTA's GSQ-RCO quantized versions across seven benchmarks including AIME25, GPQA-D, MMLU-Pro, IFEval, LiveCodeBench v6, and Aider. The team failed to reproduce ISTA's results; ISTA claimed 100% accuracy recovery on AIME25, but actual measurements show only about 91.88%. The article also discusses the significance of KLD as a quantization quality metric: it reflects output distribution fidelity but cannot replace task scores. For deployment, the model can run on 128GB memory devices, and with engram offloading can go as low as 64GB. The author reminds readers that all data is vendor self-tested, and independent reproduction plus your own small-scale validation are the keys to choosing a quantization scheme.