Breaking the 1.58-Bit Barrier: How Ternary LLMs Approach the Information-Theoretic Limit

The Quantization Frontier of Ternary LLMs

The inference cost of large language models depends heavily on how weights are stored and computed. In recent years, the field has pushed aggressively into ultra-low-bit quantization, with ternary weight schemes — constraining each weight to one of three values: -1, 0, or +1 — emerging as a promising direction.

From an information-theoretic perspective, the theoretical entropy of three equally probable values is log₂(3) ≈ 1.585 bits. This is where the "1.58-bit" figure comes from: it represents the theoretical lower bound for storing a single ternary weight under the assumption of a uniform distribution. Works like BitNet have used this number as a landmark metric, demonstrating that ternary models can achieve dramatic compression while maintaining performance close to full-precision models.

Quantization is the technique of mapping model weights from high-precision floating-point representations (e.g., FP32, BF16) to low-bit integer formats. INT8 quantization is already widely used in production, while INT4 and lower require more careful calibration and compensation strategies. Ternary quantization sits at the extreme end of this spectrum: with only three discrete values, matrix multiplication degrades to addition and subtraction, theoretically eliminating multipliers entirely — a significant advantage for hardware efficiency. BitNet b1.58, proposed by Microsoft Research in 2024, argues for introducing ternary constraints during training (rather than post-training quantization), making models naturally suited for ultra-low-bit inference from the ground up. This sparked widespread attention and debate around the "1.58-bit" metric.

Why 1.58 Bits Is Not the True Lower Bound

The 1.58-bit figure rests on an implicit assumption: that all three values appear with equal probability. But in models trained in practice, weight distributions are rarely uniform — a large fraction of weights tend toward zero, creating natural sparsity.

Once the distribution deviates from uniform, the actual information content drops below log₂(3). In other words, 1.58 bits is the entropy upper bound for the worst case, not an inevitable cost for real-world models. The paper titled Breaking the 1.58-bit Barrier for Ternary LLMs zeroes in on exactly this point: by exploiting the skewness of weight distributions and pairing it with more efficient entropy coding schemes, it becomes possible to push the average bits-per-weight below 1.58.

The Compression Room Sparsity Provides

When the proportion of zero-valued weights in a model is significantly elevated, techniques such as arithmetic coding, run-length encoding, or other probability-based compression methods can bring the average code length close to the distribution's true empirical entropy. For a ternary weight matrix dominated by zeros, the empirical entropy can easily fall in the 1.0–1.3 bit range, effectively "breaking" the nominal 1.58-bit barrier.

Arithmetic Coding is a lossless compression method that approaches the theoretical entropy limit by encoding an entire message sequence as a single fractional value within an interval, rather than assigning fixed-length codes to individual symbols. Compared to Huffman coding, arithmetic coding more precisely approximates a source's true entropy — especially when symbol probabilities are highly skewed. Run-Length Encoding (RLE), on the other hand, is specifically designed for sequences with long runs of repeated values, making it particularly effective for sparse weight matrices where zeros cluster together. Used in combination, RLE can first compress consecutive zero runs, while arithmetic coding handles the remaining non-zero symbol sequences — achieving compression rates close to the empirical entropy on ternary weights with high zero density.

Engineering Implications and Practical Constraints

The value of this line of research extends beyond theoretical bit counts — it has direct implications for on-device deployment. Lower average bits-per-weight means smaller model files, reduced memory bandwidth pressure, and the real possibility of running large models on mobile devices and edge hardware.

That said, it's important to distinguish between storage compression and computational efficiency. Entropy coding reduces static footprint on disk or in memory, but actual inference still requires decoding weights back into a computable form. If decoding overhead is too high, it may negate the bandwidth savings entirely. Real-world deployment requires balancing compression ratio, decoding speed, and hardware support.

Memory bandwidth is one of the primary bottlenecks in large model inference latency. On GPUs or NPUs, compute units are often stalled waiting for data to load rather than being limited by arithmetic throughput — a phenomenon known as the "Memory Wall." Reducing weight bit-width therefore directly lowers the amount of data that must be transferred from HBM or DRAM per inference pass, cutting memory access latency. However, the variable-length bitstreams produced by entropy coding are structurally at odds with the fixed-length, aligned memory layouts preferred by hardware SIMD instructions. Current mainstream approaches to resolving this tension include: encoding weights in independent blocks to support random access, using dedicated decompression hardware units (e.g., NVIDIA's NVComp), or designing custom inference kernels tailored to sparse ternary weights — preserving compression gains while minimizing decode latency.

A Redefinition of What "Limits" Mean

The central insight of this discussion is that so-called "barriers" are always a function of the assumptions we make. When we shift from assuming a uniform distribution to modeling the actual sparse distribution found in real weights, what once seemed like an impenetrable 1.58-bit floor turns out not to be a hard limit at all.

For researchers and engineers focused on model compression and efficient inference, this is a reminder to distinguish between the theoretical entropy upper bound and the actual information content of empirical distributions — and to treat encoding efficiency as an integral part of holistic system design.

Note: This article is based on a discussion thread on Hacker News. The original post contains limited detail; for specific technical details and experimental data, please refer to the relevant papers directly.