Q8_K_XL vs. MXFP4 Naming Debate: The Quantization Truth That FP8 ≠ Q8_0

Introduction: A Technical Debate Over Quantization Naming

In the local LLM deployment and quantization community, discussions about quantization format naming never cease. Recently, around the quantization files for DeepSeek series models (DS4), a frequently asked question emerged: why use a name like Q8_K_XL instead of simply adopting MXFP4?

This seemingly simple naming question touches on the fundamental differences between FP8, MXFP4, Q8_0, and other numerical formats, as well as implementation limitations of the llama.cpp inference framework. This article will dive deep into the distinctions between these quantization formats and clarify a widely held but inaccurate misconception — that "FP8 is equivalent to Q8_0, and both are lossless."

DS4's Mixed Quantization Architecture: Why It's Not a Single Format

The DeepSeek series models (DS4) adopt a mixed-precision quantization strategy by design, rather than using a single format across all layers:

• MoE (Mixture of Experts) layers: Use MXFP4 format
• Most other layers: Use FP8 format

This hybrid approach has clear engineering rationale: MoE layers contain a massive number of parameters and are the primary contributor to model size, so using more aggressive 4-bit (MXFP4) compression significantly reduces storage and VRAM usage. Meanwhile, layers more sensitive to precision (such as attention, embedding, etc.) retain FP8's higher-precision representation, maintaining overall output quality.

Why MoE Architecture Is the Size Bottleneck

MoE (Mixture of Experts) is a conditional computation architecture whose core idea is to split the model's feed-forward network (FFN) into multiple parallel "expert" sub-networks, with a gating network (Router) selectively activating only a few experts during each inference pass. The DeepSeek-V2/V3 series models heavily employ this architecture — for example, DeepSeek-V3 has 256 experts but activates only 8 at a time. This means that although the total parameter count is enormous (e.g., 671B), the actual computation per inference is far less than an equivalently-sized dense model. However, all expert weights still need to be stored in VRAM or RAM, making MoE layers the dominant contributor to model size — typically accounting for over 80% of total parameters. This is precisely why applying more aggressive 4-bit compression to MoE layers yields significant storage benefits.

MXFP4: The Elegant Design of Microscaling Floating Point

MXFP4 (Microscaling FP4) is a data format defined in the Microscaling (MX) specification released by the Open Compute Project (OCP) in 2023. Unlike traditional per-tensor quantization, MX formats employ a "block-level shared exponent" strategy: a group of elements (typically 32) share a single 8-bit scale factor, while each element itself is represented by only a 4-bit micro floating-point number (1 sign bit + 2 exponent bits + 1 mantissa bit). This design preserves the dynamic range advantages of floating-point numbers even at extremely low bit-widths, offering noticeably better representational capacity compared to pure integer INT4 quantization. NVIDIA's Blackwell architecture (B100/B200) already provides native hardware support for MXFP4 matrix operations, enabling hardware acceleration for both training and inference. DeepSeek chose MXFP4 for MoE layers precisely because of its favorable precision-to-size tradeoff at extremely low bit-widths.

The Fundamental Reason It Can't Simply Be Called MXFP4

The key issue is inference framework compatibility. The original technical notes explicitly state:

llama.cpp doesn't have native FP8 & Q8_0 != FP8.

In other words, the llama.cpp inference framework does not natively support the FP8 format. When models are ported to the llama.cpp ecosystem, the original FP8 layers cannot be used directly and must be converted to quantization types the framework supports (such as Q8_0, BF16, etc.).

Historical Context of llama.cpp and the GGUF Ecosystem

llama.cpp is an open-source project initiated by Georgi Gerganov in March 2023, with the goal of efficiently running large language model inference on pure CPU (and later GPU). It introduced GGUF (GPT-Generated Unified Format) as the model file format, defining a complete set of quantization types (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, F16, F32, etc.), each with different bit-widths, blocking strategies, and scale factor precisions. llama.cpp's design philosophy centers on accessibility for consumer-grade hardware, so its quantization type system is primarily integer-based, with deep optimizations for ARM NEON, AVX2, AVX-512, and other instruction sets. Because of this historical architectural decision, llama.cpp currently has not implemented native FP8 data type compute kernels — when the community needs to port FP8-native models (like DeepSeek DS4) to GGUF format, FP8 weights must be converted to types the framework already supports, which creates the conversion precision issues discussed in this article.

Since the model is no longer "pure MXFP4," using MXFP4 as the name would be seriously misleading. Therefore, the community adopts names like Q8_K_XL and Q4_K_XL to reflect the actual numerical composition after conversion.

Deconstructing Quantization Names: The Real Composition Behind K_XL

The core value of naming lies in accurately describing what formats a quantization file is actually assembled from. Here's a comparison of two key approaches:

Q8_K_XL = MXFP4 + BF16 (100% Lossless)

In the Q8_K_XL approach:

• MoE layers: Retain the original MXFP4
• Other layers: Originally FP8 layers are upcast to BF16 (16-bit brain float)

Why FP8→BF16 Is Mathematically Lossless

BF16 is a 16-bit floating-point format proposed by the Google Brain team, containing 1 sign bit, 8 exponent bits, and 7 mantissa bits. Compared to standard FP16 (1+5+10), BF16 sacrifices mantissa precision (7 bits vs. 10 bits) but gains the same number of exponent bits and dynamic range as FP32 (approximately ±3.4×10³⁸). This means BF16 rarely encounters the overflow issues common with FP16. Since FP8 E4M3 has a dynamic range of only ±448 and just 3 mantissa bits of precision, while BF16 has 7 mantissa bits and far exceeds FP8's dynamic range, the conversion from FP8 to BF16 is a precision upgrade — BF16 can exactly represent every value representable by FP8 E4M3 (the reverse is not true). This is the mathematical basis for why the technical notes mark FP8→BF16 as "100% lossless."

The tradeoff is larger file size, since 8-bit FP8 is expanded to 16-bit BF16. But for users pursuing maximum fidelity, this is the safest choice.

Q4_K_XL = MXFP4 + Q8_0 (96% Top-1% Agreement)

The Q4_K_XL approach prioritizes size compression:

• MoE layers: Same MXFP4
• Other layers: Compressed to Q8_0

Technical testing shows this approach achieves "96% same" top-1% prediction agreement, meaning the model's top prediction matches the original model in the vast majority of cases, but with approximately 4% divergence.

What the Top-1% Agreement Metric Means

Top-1% agreement (or same rate) is a metric used in quantized model quality evaluation. It selects the highest-probability tokens from the model's output logits (typically top-1 or top-k) and compares whether the quantized model and original model produce the same top prediction given identical inputs. 96% top-1% agreement means: across a large number of test tokens, the quantized model's highest-probability prediction exactly matches the original model at 96% of positions, with divergence at 4% of positions. This divergence can produce cumulative effects in long-text generation — because in autoregressive generation, each step's output becomes the next step's input, and a single token difference can lead to a completely different generation path. However, for most practical scenarios (such as conversation, summarization), 4% token-level divergence typically doesn't significantly affect user-perceived output quality.

This clearly demonstrates: Q8_0 is not strictly lossless. It is an 8-bit integer quantization format with fundamental differences from floating-point formats in numerical representation, and the conversion process introduces small but measurable precision loss.

Core Clarification: What's Actually Different Between FP8 and Q8_0

The most common misconception in the community is equating FP8 with Q8_0. The original technical notes give the most direct response:

Folks saying FP8==Q8_0 as lossless are wrong.

Many people see "both are 8-bit" and assume they're equivalent. But in reality, these two formats have fundamental differences in their numerical representation principles:

FP8: A Floating-Point Format

FP8 is a floating-point format, typically with E4M3 or E5M2 variants, containing sign bits, exponent bits, and mantissa bits, possessing the inherent dynamic range characteristics of floating-point numbers. It can better represent extremely large or extremely small values.

FP8 E4M3 contains 1 sign bit, 4 exponent bits, and 3 mantissa bits, with a representable value range of ±448 and precision equivalent to approximately 3 significant decimal digits, suitable for storing weights and activations during forward inference. E5M2 contains 1 sign bit, 5 exponent bits, and 2 mantissa bits, with a larger dynamic range (±57344) but slightly lower precision, commonly used for gradient representation. Compared to FP16 (±65504, precision of approximately 3-4 significant decimal digits), FP8 E4M3's dynamic range and precision are both significantly reduced, but compared to integer quantization it still retains the inherent non-uniform distribution characteristic of floating-point numbers — "denser near zero." NVIDIA H100 and subsequent GPUs provide native FP8 Tensor Core support, meaning FP8 inference on this hardware incurs no additional dequantization overhead.

Q8_0: An Integer Quantization Format

Q8_0 is an integer quantization scheme used by llama.cpp that maps a group of weights to an 8-bit integer range through a scale factor, essentially performing block-wise linear quantization. It has notable limitations in dynamic range compared to floating-point formats.

Its specific implementation works as follows: weights are grouped into fixed-size blocks (typically 32 elements), a FP16 scale factor is computed for each block (scale = max(abs(block)) / 127), then all values within the block are divided by the scale and rounded to the integer range [-127, 127]. During dequantization, integer values are simply multiplied by their corresponding scale to approximately recover the original floating-point values. This method is essentially piecewise uniform quantization — within each block, quantization intervals are equally spaced, unable to achieve higher resolution near zero as floating-point formats do. For layers where weight distributions exhibit long-tail characteristics, Q8_0 may produce significant rounding errors at extreme values. Additionally, every 32 elements require an extra 2-byte scale to be stored, making the effective average bit-width per element approximately 8.5 bits.

Although both use 8 bits to store a single value, their numerical distributions and representational capabilities are completely different. Converting FP8 to Q8_0 is not an "equivalent copy" but a lossy remapping. The irreconcilable structural difference between floating-point's non-uniform quantization intervals (denser near zero) and integer quantization's uniform intervals is particularly pronounced when weight distributions are non-uniform.

Why Naming Precision Matters

Precisely because of these differences, Q8_K_XL (using BF16, lossless) and Q4_K_XL (using Q8_0, 96% agreement) need to be clearly distinguished. Naming is not just a label — it's a quality promise to users: it tells you exactly how much precision this quantized version retains relative to the original model.

How Local Deployment Users Should Choose Quantization Versions

This naming discussion has several direct implications for developers actually using large models:

1. Choose versions based on task precision requirements: If your task is extremely precision-sensitive (e.g., code generation, mathematical reasoning), prioritize Q8_K_XL lossless versions; if VRAM is limited and slight quality fluctuations are acceptable, Q4_K_XL is the more economical choice.

2. Don't blindly trust "8-bit means lossless": 8-bit only describes bit-width; whether it's floating-point or integer quantization directly determines precision performance.

3. Understand framework limitations before making decisions: llama.cpp's lack of native FP8 support is a real constraint when converting FP8-native models like DeepSeek to GGUF format, and explains why quantization files use mixed naming conventions.

4. Pay attention to the size-precision tradeoff: Taking the DeepSeek-V3 671B model as an example, MoE layers account for the vast majority of parameters. Q8_K_XL and Q4_K_XL are identical in storage for MoE portions (both MXFP4), differing only in non-MoE layers — BF16 is twice the size of FP8, while Q8_0 is roughly equivalent to FP8 in size. Therefore, the actual total file size difference depends on the proportion of non-MoE layers.

Conclusion

A seemingly trivial naming question reflects the rigor that quantization technology demands. Q8_K_XL isn't called MXFP4 because it honestly reflects the "MXFP4 + BF16" hybrid composition, rather than obscuring the format changes that occur during the llama.cpp conversion process.

As large model quantization becomes increasingly widespread, insisting on naming precision is precisely the foundation for ensuring users make correct technical choices. Remember the key takeaway: FP8 is not equal to Q8_0, and 8-bit doesn't necessarily mean lossless.