VLCQ: How Variable-Length Coding Breaks Through the INT8 Quantization Bottleneck

When deploying deep learning models to edge devices and mobile platforms, quantization is almost an unavoidable step. Among various quantization approaches, Post-training Quantization (PTQ) has drawn sustained attention because it requires neither the original training data nor model retraining, making it the lowest-cost option for engineering deployment. However, existing methods share a hidden assumption: to maintain accuracy close to FP32, you basically need INT8 fixed-point integers; once you compress the bit width further, performance drops noticeably.

A recent paper titled "VLCQ: Post-training quantization for deep neural networks using variable length coding" proposes a different approach: abandon the fixed-point encoding constraint and instead use variable-length coding to explore the quantization space, thereby achieving a better balance between higher compression rates and better accuracy without accessing training data or fine-tuning the model.

The Predicament of Fixed-Point Quantization

The logic of traditional fixed-point quantization is straightforward: map floating-point weights and activations onto an integer grid of fixed bit width—for example, INT8 corresponds to 256 discrete levels. The lower the bit width, the fewer representable value levels, and the greater the quantization error.

The paper points out that current post-training quantization methods capable of approaching baseline performance typically rely on INT8 fixed-point integers. This creates an awkward situation—if you want high accuracy, you must accept the storage and computation overhead of 8 bits; if you want higher compression, you must sacrifice model performance. For vision models with hundreds of millions of parameters, this contradiction is especially pronounced.

The divide between Post-training Quantization (PTQ) and Quantization-Aware Training (QAT) is worth elaborating on. QAT simulates quantization errors during the training phase, allowing model parameters to actively adapt to low-bit-width representations, thus typically achieving better accuracy—but at the cost of requiring full training data, a training pipeline, and several times the compute of inference. PTQ, on the other hand, directly maps weights and activations after model training is complete, using only a small calibration set (or even no data at all) to estimate value ranges, making engineering costs far lower. VLCQ belongs to the category of fully data-free PTQ, meaning it doesn't even need calibration data, making it particularly suitable for scenarios where training data is inaccessible due to privacy, copyright, or collection costs—such as medical imaging or automotive perception models.

VLCQ's Core Idea: Replacing Fixed-Point Mapping with Variable-Length Coding

VLCQ's key innovation lies in "relaxing" the fixed-point encoding constraint. It no longer requires all quantized values to share the same fixed bit width, but instead represents quantized values through variable-length encoding.

The significance of this shift is that densely distributed value regions can be represented with finer encoding, while sparse regions use shorter codes, overall staying closer to the true distribution characteristics of weights and activations. Since it doesn't rely on training data, VLCQ remains purely within the post-training quantization category, adding no cost for data acquisition or model fine-tuning.

Variable-Length Coding (VLC) originates from entropy coding ideas in information theory, with the most typical examples being Huffman coding and arithmetic coding: frequently occurring symbols are assigned short codewords, while infrequent ones get long codewords, thereby bringing the average code length close to the source entropy. VLCQ transfers this idea to the representation of quantized values—the actual distributions of weights and activations are often heavy-tailed or approximately Gaussian, with a few intervals being extremely dense and most being sparse, yet fixed-point quantization allocates bits equally to each level, causing waste. Variable-length coding allows dense intervals to be finely characterized with more codewords and sparse intervals to be roughly represented with fewer codewords, theoretically reducing quantization error at the same average bit width. The cost is that codeword lengths are no longer aligned, and decoding requires variable-length prefix parsing or table lookup, introducing additional complexity for hardware implementation.

Experimental Performance: Dual Advantages in Compression Rate and Accuracy

The paper conducts extensive experiments on three types of tasks: image classification, segmentation, and object detection, with conclusions pointing to improvements at two levels.

In per-channel quantization, VLCQ surpasses FP32 accuracy and Piece-Wise Linear Quantization (PWLQ) on most evaluated models. Compared to FP32, it achieves up to 6x compression; compared to PWLQ, up to 1.7x compression. When prioritizing compression more heavily and tolerating limited accuracy loss, VLCQ can achieve up to 12.25x compression compared to FP32, with performance degradation kept within 4%.

In per-tensor quantization, VLCQ performs comparably to Data-Free Quantization (DFQ), but offers greater flexibility across different tasks and models, allowing more freedom to push down the bit rate.

You might not have noticed that VLCQ's ability to surpass FP32 accuracy in per-channel scenarios is quite counterintuitive—quantization typically introduces errors, but the representational flexibility brought by variable-length coding may act somewhat like regularization on certain models, actually improving generalization performance. The paper doesn't delve deeply into the mechanism behind this, which is a direction worthy of follow-up research.

Per-channel and per-tensor are two fundamental choices in quantization granularity. Per-tensor quantization shares a single set of scaling factors and zero points across the entire tensor, making it simplest to implement and most hardware-friendly, but when different channels within a tensor have vastly different value ranges, small-range channels get "squeezed" by large-range channels, significantly increasing quantization error. Per-channel quantization calculates scaling parameters separately for each output channel, typically yielding higher accuracy, at the cost of needing to index parameters by channel during storage and computation, which is less friendly for operator fusion and SIMD acceleration. VLCQ's more pronounced gains in per-channel scenarios suggest it can better exploit fine-grained distribution information; its parity with DFQ in per-tensor settings also reflects that when granularity is coarse, the gains from encoding flexibility are offset by distribution aliasing.

Implications for Edge Deployment

VLCQ's value isn't just in the numerical improvement in compression rate. It provides a signal: the bottleneck of post-training quantization may not be "insufficient bit width," but rather encoding methods that are too rigid. Fixed-point integers facilitate hardware acceleration, but variable-length coding's advantage in storage efficiency opens new space for scenarios with extremely constrained resources.

Of course, variable-length coding also brings increased encoding/decoding complexity, and the paper doesn't discuss in detail the actual hardware-friendliness during inference. If decoding overhead cancels out storage gains, the cost-effectiveness of engineering deployment needs to be reassessed.

PWLQ (Piece-Wise Linear Quantization) is a relatively representative comparison baseline in the PTQ field. Its approach is to divide the value domain into several intervals, each with its own independent linear mapping—essentially piecewise approximation of nonlinear distributions, still an improvement within the fixed-point framework. DFQ (Data-Free Quantization), on the other hand, emphasizes complete detachment from data, reducing quantization error through analytical means such as weight equalization and bias correction, serving as a benchmark method for data-free PTQ. VLCQ differs from both in that: PWLQ merely makes fixed-point mapping finer, DFQ merely makes calibration cleverer—neither escapes the premise of "fixed code length"; VLCQ changes the encoding mechanism itself, which is why it can achieve order-of-magnitude breakthroughs in compression rate, and precisely why it introduces the variable-length decoding overhead that PWLQ and DFQ lack.

Summary

By introducing variable-length coding, VLCQ achieves a breakthrough beyond fixed-point constraints within the post-training quantization framework. Without requiring training data or fine-tuning, it demonstrates superior compression-accuracy trade-offs over PWLQ and DFQ on image classification, segmentation, and detection tasks. For researchers and engineers focused on model compression and edge deployment, this is a direction worth tracking.

One-line Summary

VLCQ replaces fixed-point mapping with variable-length coding to achieve higher-compression, data-free, fine-tuning-free post-training quantization.