Q4_K_M

In GGUF quantization naming, Q4_K_M denotes a 4-bit quantization, Medium-precision variant using the K-Quant grouped quantization method

The Qwen2.5:7B model is approximately 4 GB in size after Q4 quantization.

Q4_K_M中的K代表k-quants方案，M代表中等质量级别，是llama.cpp引入的混合量化策略

Community rule of thumb holds that Q4_K_M (4-bit medium-precision quantization) offers the best balance between performance and file size in most scenarios.

GGUF格式中的Q4_K_M采用分组量化（Group Quantization）策略，每组独立计算缩放因子和零点

The perplexity loss of Q4_K_M quantization compared to FP16 is typically within 1%–3%

With flash attention disabled, Q4_K_M outputs are bitwise identical at ubatch sizes that are multiples of 4 and not at other sizes, indicating that llama.cpp's CPU path is not batch-invariant across all batch sizes.

In a comparison of Qwen3-0.6B in BF16 versus Q4_K_M, Q4_K_M changed the top-1 token at 11.6% of positions, with an average KL divergence of 0.067 and perplexity rising from 7.83 to 8.53.

At the is_prime function definition, the BF16 version has a 97.9% probability of outputting False, whereas Q4_K_M has a 79.1% probability of outputting True.

Layer-by-layer denoising experiments between Q4_K_M and BF16 show KL decreasing smoothly from layer 0 to layer 27, with no single layer carrying all the difference.

At the is_prime token, after writing BF16's residual into Q4_K_M, that token's KL dropped from 0.48 after layer 6 to 0.04 after layer 9; the reverse write reproduced the flip from about layer 12, indicating the decisive difference forms between layers 6 and 12.

Across 12 blocks and 1282 scored positions, when Q4_K_M recovers BF16, attention eliminates 0.030 (95% CI 0.019–0.041) and MLP eliminates 0.036 (0.024–0.048), with the paired difference crossing zero so the two are indistinguishable; when injecting Q4_K_M into BF16, MLP adds more (0.014, 0.005 to 0.023).

Q4_K_M量化大约表示每个权重4位，并对异常值采用特定方案，在内存受限的机器上通常是甜点选择。

The comparison baseline Q4_K_M GGUF version runs at about 42 tok/s on llama.cpp on the same card (RTX 3090); HyperQwen is about three times faster in real chat and about six times faster in citation scenarios.

W4A16 quantisation refers to a mixed-precision scheme in which weights are stored in 4-bit and activations remain 16-bit; compared with Q4_K_M it retains a higher-precision activation path, and combined with dedicated CUDA kernels such as marlin it can achieve higher throughput on NVIDIA graphics cards, but VRAM usage is more rigid and kernel support requirements are stricter.

Q4_K_M is a GGUF quantization format widely used in the llama.cpp community, in which weights are compressed to about 4 bits using the medium tier of the k-quant mixed quantization scheme, striking a compromise between size and precision.

The author claims Janas beats llama.cpp on all 13 compared models, under the same test conditions of the same laptop and Q4_K_M quantization.

Letter suffixes in quantized version names represent different quantization strategies; the K series uses mixed precision to retain higher precision for critical layers and generally outperforms earlier uniform quantization schemes at the same size.

The author recommends Q4_K_M or Q5_K_M quantization for code assistance, and Q5_K_M or Q6_K for math and logic tasks to ensure reasoning quality.

Q4_K_M is generally regarded as the best balance between quality and size, retaining approximately 95% of the original model's capability while compressing the size to about one-quarter of FP16.