Demystifying LLM Quantization Through Animation: From absmax to BitNet

Using 8 weights, it connects the quantization trade-off chain of absmax, outliers, block-wise scaling, and BitNet.
The article starts from the fact that a 70B model at 16-bit requires 140 GB of VRAM, explaining that quantization is a precision game of lossy mapping within a limited numerical space. Through 8 concrete weights, it demonstrates absmax: the scaling factor is determined by the global maximum, and rounding error erases the distinguishability of close weights. A single outlier can stretch the step to about 2, zeroing out all small weights; Dettmers et al. (2022) confirmed this phenomenon in real LLMs. Engineering solutions turn to block-wise scaling: llama.cpp's Q4_K scales independently with 32 weights per block, equivalent to about 4.5 bits, reducing a 70B model to about 39 GB. BitNet b1.58, meanwhile, restricts weights to -1/0/+1 from the training stage, requiring only addition and subtraction for inference, but cannot be directly applied to existing models.
Large model quantization is the key technology that lets a 70B-parameter model run on consumer-grade GPUs. A creator on Reddit used a 7-minute animation, with 8 concrete weights as examples, to connect absmax mapping, rounding error, outliers, block-wise scaling, and 1.58-bit BitNet into a complete technical chain. This article, based on the video content, outlines the core logic and engineering trade-offs of quantization.

The Core Contradiction Quantization Must Solve: Memory vs. Precision
The video opens with a concrete problem: a 70B model at 16-bit precision requires 140 GB of VRAM, while gaming GPUs typically have only 24 GB or 32 GB. To run it locally, the number of bits each weight occupies must be reduced.
But reducing bits means the number of representable values drops sharply. An 8-bit integer has only 256 possible values, a much narrower dynamic range than floating point. The essence of quantization is preserving as much of the model's key behavioral information as possible within a limited numerical space. This is not simple compression, but a precision game of lossy mapping.
The first step to understanding quantization is to see where error comes from in a single mapping process.
absmax Mapping and Rounding Error: The First Lesson in Quantization Precision
The absmax method uses the maximum absolute value to determine the scaling factor. In the video, the largest weight is 10.8, mapped to the upper limit of an 8-bit integer, 127, giving a scaling factor of about 11.76. The weight 5.47 is stored as 64, and after dequantization becomes 5.44; the difference of 0.03 is the rounding error.
The error itself is not frightening; what is frightening is that it erases the distinguishability of weights. Two close weights, 3.08 and 3.02, both become 36 after quantization, and the model can no longer distinguish them. This information loss accumulates and amplifies at low bit widths, directly affecting model output quality.
The problem with absmax is that the scaling factor is determined by the global maximum, and a few extreme values can hijack the entire value distribution.
Scaling factor and zero-point are the two basic parameters for understanding quantization mapping. The scaling factor determines the proportional relationship between the floating-point interval and the integer interval, while the zero-point marks the position in integer space corresponding to floating-point zero. Symmetric quantization (such as absmax) usually sets the zero-point to 0, with positive and negative weights sharing the same scaling factor; asymmetric quantization allows the zero-point to shift, to better cover asymmetric distributions. Dequantization uses float ≈ scale × (int - zero_point) to restore, so the storage precision and rounding method of the scaling factor both introduce additional quantization noise.
Outliers: The Fatal Weakness of Global Scaling
The video uses a set of comparative data to show the destructive power of outliers: when most weights are below 0.6 but a single outlier of 256 appears, the quantization step is stretched to about 2. As a result, all small weights are rounded to 0, and almost all detailed information in the model is lost.
This is not a theoretical assumption. Dettmers et al. (2022) found such outliers in real LLMs. They are few in number but large in magnitude, posing a structural challenge to global scaling schemes.
The existence of outliers shows that covering all parameters with a single unified scaling factor does not work in real models. Engineering solutions turn to finer-grained scaling strategies.
LLM.int8() is a representative solution proposed by Dettmers et al. after discovering outliers. Its core idea is mixed-precision decomposition: most weights still use 8-bit matrix multiplication, but outlier feature dimensions whose magnitude exceeds a threshold are extracted separately and computed in 16-bit floating point. This preserves most of the computational throughput while preventing outliers from contaminating global scaling. The work also observed that outliers tend to concentrate in a few fixed channels and become more pronounced as model size increases, providing empirical evidence for subsequent designs based on per-channel or per-block scaling.
Block-wise Scaling and Q4_K: An Engineering Compromise
Since global scaling is dragged down by outliers, divide the weights into small blocks and compute a scaling factor independently for each block. llama.cpp's Q4_K uses 32 weights per block, ultimately achieving about 4.5 bits per weight, reducing the memory footprint of a 70B model to about 39 GB.
This number is worth noting: 4.5 bits is not an integer bit width, but an equivalent value after combining intra-block scaling, quantization parameters, and data bits. The cost of block-wise scaling is the need to store additional scaling factors for each block, but in exchange, the impact of outliers is confined locally and does not contaminate the entire tensor.
Block-wise scaling is the foundation of current mainstream quantization schemes, but it is still an approximation of weights after training. Another route changes the representation of weights starting from the training stage.
GGUF and K-quant are the file format and algorithm family in the llama.cpp ecosystem that carry block-wise quantization. GGUF defines the on-disk layout of quantized tensors, while K-quant (such as Q4_K and Q5_K) introduces two-level scaling within blocks: a super-block holds coarse-grained scaling, and sub-blocks hold fine-grained scaling, trading a small amount of extra metadata for a smoother error distribution. Choosing Q4_K rather than the lower Q2_K is usually a trade-off between perplexity and VRAM—the more aggressive the quantization, the lower the equivalent bit width, but the more obvious the degradation on long-tail knowledge.
BitNet b1.58: Restructuring Weights from the Training Stage
BitNet b1.58 restricts each weight to three values: -1, 0, and +1, equivalent to about 1.58 bits. The key difference is that it is not post-training quantization; instead, the model adapts to this representation during training.
The computational advantage of this design is directly reflected at the hardware level: a layer requires only addition and subtraction, and multipliers can be greatly simplified. The video points out that this ternary representation makes inference computation extremely lightweight.
But the cost is equally clear: the model must adopt this architecture from the training stage, and it cannot be directly applied to existing models. BitNet represents a paradigm shift in quantization thinking, from compressing existing weights to designing models that are inherently low-bit.
Quantization-aware training (QAT) versus post-training quantization (PTQ) is the key to understanding BitNet's positioning. PTQ approximates already-trained weights, with low cost but uncontrollable error; QAT simulates quantization rounding in the training forward pass, allowing gradients to adapt to low-bit representations, usually significantly narrowing the accuracy gap. BitNet b1.58 is a more aggressive QAT variant: it writes the quantization operator directly into the architecture and trains ternary weights from scratch. This also explains why it cannot be directly applied to existing models—the representation of weights has already changed the model's trainable space.
The Logic of Choosing a Quantization Route
Putting the threads from the video together, a clear trade-off map emerges. absmax is simple but constrained by outliers; block-wise scaling confines error locally through fine-grained control and is the mainstream choice for current local deployment; BitNet restructures from the training side in exchange for extreme computational efficiency, but requires retraining the model.
Each route corresponds to different use cases. To reduce VRAM usage on existing open-source models, block-wise quantization is the pragmatic path; to pursue extreme inference efficiency and have training resources, the ternary architecture offers another possibility. The core proposition of quantization is always the same: within a limited numerical space, how to minimize the loss of model behavior. Understanding this proposition is more important than remembering a specific scheme.
One-line Summary
Using 8 weights as examples, it connects the quantization trade-off chain of absmax, outliers, block-wise scaling, and BitNet.
Paragraph Summary
The article starts from the fact that a 70B model at 16-bit requires 140 GB of VRAM, explaining that quantization is a precision game of lossy mapping within a limited numerical space. Through 8 concrete weights, it demonstrates absmax: the scaling factor is determined by the global maximum, and rounding error erases the distinguishability of close weights. A single outlier can stretch the step to about 2, zeroing out all small weights; Dettmers et al. (2022) confirmed this phenomenon in real LLMs. Engineering solutions turn to block-wise scaling: llama.cpp's Q4_K scales independently with 32 weights per block, equivalent to about 4.5 bits, reducing a 70B model to about 39 GB. BitNet b1.58, meanwhile, restricts weights to -1/0/+1 from the training stage, requiring only addition and subtraction for inference, but cannot be directly applied to existing models.
Related articles

Jingle Jam 2026 Registration Opens: Lewis Uses Old Footage to Rally Volunteers for Fundraising
Jingle Jam 2026 fundraising registration is open. Lewis uses old footage edited into a new video, calling on viewers to create fundraising pages at jingle jam.tiltify.com to support eight charities. The event takes place in the first two weeks of December.

Gemini 4 Neon Hands-On Test: Can It Replace Claude Opus 5.5?
A Bilibili creator tests Gemini 4 Neon, Claude Opus 5.5, and GPT 6.2 across three coding Agent challenges—pelican animation, 3D racing, and code-generated video—to see if Gemini can replace Opus.

FBI-Warned Artex: How an AI Attack Framework With No In-House Model Pries Open Enterprise Doors
The FBI-warned open-source AI attack framework Artex has no model of its own—it plugs into Claude, GPT, and other external LLMs to drive an agent loop, turning ordinary tools like Nmap into an automated attack pipeline. This article examines three real cases—South Korean finance, ConnectWise, and Barracuda—analyzes the three drivers of speed, cost, and noise, and outlines defenses such as exposure inventory and short-lived credentials.