Compression as Intelligence: How to Compress Text to Under 1 Bit per Character

Arithmetic coding turns language model probability predictions into compression approaching the Shannon limit, revealing the deep equivalence between compression and intelligence.
Starting from the Hutter Prize challenge, the article explains how arithmetic coding converts a language model's probability predictions into near-optimal compression. The core principle is mapping each character's probability distribution onto the unit interval; the more accurate the model, the closer the required bits are to Shannon information. Engineering solutions like renormalization and ANS address precision. It further shows that minimizing cross-entropy in language model training is equivalent to minimizing bits per token, and that compression and prediction are mathematically equivalent. Accounting for decoder size, smaller models may be better; online fine-tuning can improve compression. The conclusion: compression rate captures intelligence's predictive ability, simplicity, and adaptation speed, serving as a quantifiable proxy for intelligence.
From the Shannon Limit to 1 Bit per Character: How Arithmetic Coding Turns Prediction Models into Compressors
In information theory, one long-standing challenge is how to compress text to the extreme. The Hutter Prize was established precisely for this: whoever can compress 1GB of Wikipedia the smallest wins the prize. A recent technical explainer animation reveals a surprising fact—all current winning solutions rely on the same universal technique: combining a language model with large-number arithmetic coding to achieve compression rates approaching or even below 1 bit per character.

This approach works not only for text but also for images, audio, video, and any other data type using the same idea. More intriguingly, it provides a concrete mathematical path for understanding the proposition that "compression is intelligence."
Core Principle: Mapping Probability Distributions onto the Unit Interval
Starting from Shannon Information
In the 1940s, Claude Shannon proved that the optimal number of bits for compressing a message is approximately the negative base-2 logarithm of its probability. Halving the probability adds 1 bit of information. This means a highly predictable text theoretically requires very few bits to represent. However, traditional encodings such as ASCII or Huffman coding, because they do not exploit context, consume far more than the theoretical limit in practice—a text of over 200 letters requires more than 2500 bits in ASCII, and Huffman coding can only compress it to about 1500 bits.
The problem lies in fixed encodings: each character, regardless of context, corresponds to a fixed-length bit string. Shannon's theory clearly states that compressibility depends entirely on predictability, and predictability depends heavily on context.
Shannon information content is one of the most fundamental concepts in information theory, used to quantify the "surprise" or "uncertainty reduction" carried by a random event. For a discrete random variable X, when a specific value x is observed, its self-information is defined as I(x) = -log₂ P(x), measured in bits. This definition intuitively expresses that the less likely an event (the smaller its probability), the more information it provides once it occurs. For example, flipping a fair coin, the probability of heads is 0.5, giving 1 bit of information; while flipping a biased coin with a 0.9 probability of heads gives only about 0.152 bits. Shannon further proved that the source entropy H(X) = -Σ P(x)log₂P(x) is the lower bound on average information and the theoretical limit of lossless compression. This provides a benchmark for all subsequent compression algorithms: the average code length of any encoding scheme cannot be lower than the source entropy.
Language Models as Prediction Engines
Modern language models happen to excel at predicting the probability distribution of the next character or token given context. The core idea of arithmetic coding is: map the probability distribution of each possible character onto the unit interval from 0 to 1, with higher-probability characters occupying wider intervals. As characters are input one by one, the model continuously updates its predictions, the interval keeps shrinking, and finally the entire text corresponds to an extremely narrow real-number interval—only a single real number within that interval needs to be transmitted.

The receiver holds an identical language model and starts from an empty context, decoding each character step by step: first see which character's interval the real number falls into, then use that character as the new context to continue decoding the next character. Both models must be exactly identical, otherwise decoding will diverge.
A language model is a statistical model that estimates the probability distribution of text sequences, with the core task of predicting the probability of the next word or character given a context. Modern language models are typically based on neural networks, especially the Transformer architecture, learning the statistical regularities and semantic structure of language by training on massive amounts of text. In the compression context, the language model plays the role of a "probability oracle": for any given context, it outputs a probability distribution over all possible next tokens in the vocabulary. The more accurate this distribution is (i.e., the closer it is to the true data-generating distribution), the fewer bits the arithmetic coder needs to represent the token that actually appears. Notably, the language model does not need to "understand" the text—it only needs to provide sufficiently good probability estimates; but conversely, excellent compression performance often implies that the model has captured deep regularities in the data, which is precisely the starting point of the "compression is intelligence" discussion.
How the Number of Bits Approaches the Theoretical Limit
For a real-number interval of width p, to guarantee that k bits can fall within it, we need 2^{-k} < p, i.e., k ≥ -log₂(p). This is exactly the Shannon information content. Therefore, as long as the model predicts accurately, the number of compressed bits is almost equal to the information content, plus a small overhead for message length. For extremely long texts, the length overhead is negligible, and the final compression rate approaches the Shannon limit arbitrarily closely.
Take the word "mathematics" as an example. The model considers its probability to correspond to 18 bits of information, but because the interval happens to overlap with a 10-bit binary number, it is actually encoded successfully in only 10 bits—an average of 0.91 bits per character. This kind of "luck" averages out over long texts, but overall it remains highly close to the limit.
From Theory to Practice: Engineering Implementation of Arithmetic Coding
Precision Issues and Renormalization
Directly using infinite-precision real numbers is obviously unrealistic. In 1976, Jorma Rissanen and Richard Pasco independently proposed arithmetic coding, whose core trick is to renormalize the interval to length 1 after each step while outputting the bits already determined. This allows continuous encoding with only finite precision.
Renormalization in arithmetic coding is the key mechanism for solving the finite-precision problem. In standard arithmetic coding, the encoder maintains a current interval [low, high), initially [0,1). Each time a symbol is encoded, the interval narrows according to that symbol's probability distribution. But since computers can only represent integers with a finite number of bits, the interval cannot shrink indefinitely. When the interval width falls below a certain threshold, the encoder performs renormalization: it checks whether the most significant bits of low and high are the same; if so, it outputs that bit and shifts the interval left by one position (i.e., multiplies by 2) and expands it; if there is an underflow risk (low close to 0.5 and high close to 1.0), special handling is applied using the E3 condition. This process ensures the interval always remains within the available precision range while gradually outputting the already-determined bits. The decoder performs a completely symmetric operation, reading bits from the code stream and reconstructing the interval. Renormalization enables arithmetic coding to handle messages of arbitrary length with O(log n) precision, and is the cornerstone that made arithmetic coding practical.
Asymmetric Numeral Systems (ANS)
In 2013, Jarek Duda proposed another scheme—Asymmetric Numeral Systems (ANS)—which uses a growing integer to encode messages, avoiding real-number precision issues and running faster. Both methods can convert any predictor into a compressor and are already widely used in modern compression tools.
Asymmetric Numeral Systems (ANS) was proposed by Polish scholar Jarek Duda in 2009 and widely adopted after 2013. Unlike arithmetic coding, which maintains a real-number interval, ANS encodes the entire message using a growing integer state, mapping symbols to new states through reversible "state transitions." Its core advantage is that encoding and decoding can proceed symbol by symbol without the interval underflow and complex renormalization logic of arithmetic coding, resulting in faster speed and better parallelism. ANS has several variants, such as rANS (range ANS) and tANS (table ANS), suited to different probability distributions and hardware environments. Today, ANS has been integrated into modern compression tools such as Zstandard and LZMA2, and applied in image and video coding standards (e.g., JPEG XL). From a theoretical perspective, ANS can likewise convert any prediction model into a compressor, with a compression rate approaching the Shannon limit just like arithmetic coding—only the implementation path differs.
A New Perspective on Language Model Training: Compression as the Objective
The Essence of Cross-Entropy Loss
The compressed size can be written as negative log-probability, and probability is the sum of the logarithms of each predicted probability's product. Each term corresponds to the model's "surprise" at the true next character. Minimizing cross-entropy during training is actually minimizing the average number of encoded bits per token.

This provides a psychologically clearer framework: the goal of pretraining is not "predict the next token" but "become the best text compressor." Cross-entropy loss is no longer an arbitrary choice—it directly corresponds to the number of bits.
The Trade-off Between Model Size and Compression Rate
Using a small model (e.g., NanoGPT with 3.7M parameters) to encode a text yields about 2 bits per character on average, twice Shannon's estimate. Switching to the open-source model Qwen with 500 million parameters brings the average down to below 1 bit per character. For highly predictable text (such as the preamble to the US Constitution), the later parts add almost no cost—only 10 bits are needed, two orders of magnitude better than Huffman coding.
However, this raises a reasonable doubt: the model itself may have already memorized these texts—is the compression rate "cheating"? The scoring rules of the Hutter Prize already account for this—the compressed result must include the size of the decoding program. Therefore, even if a large model predicts accurately, its parameter size is counted in the total cost.
The Deep Connection Between Compression and Intelligence
The Equivalence of Prediction and Compression
Arithmetic coding demonstrates the conversion from predictor to compressor. Conversely, any compressor can also be used for prediction: for a candidate string, compare the increase in compressed length—the smaller the increase, the higher the probability. Thus, compression and prediction are mathematically equivalent.
Miniaturization and Generalization Ability
Between two models with the same predictive power, the one with fewer parameters is considered to "understand" the data better because it relies less on memorization. Under the compression framework, after accounting for decoder size, a small model may achieve a better overall compression rate. This is consistent with the principle of "Occam's razor": simpler theories are often closer to the essence.

Fast Learning and Online Adaptation
The compression process can also incorporate online learning: the encoder and decoder synchronously fine-tune the model parameters to adapt to a specific text. As long as the update rule is deterministic, both parties can stay in sync. In this way, the compression rate depends not only on static predictive ability but also on how quickly the model learns new content. The faster the learning, the better the compression. This echoes the characteristic of "rapid adaptation to new domains" in intelligence.
Conclusion: Compression as a Proxy for Intelligence
From Shannon's theory to arithmetic coding to language model training, a clear thread emerges: compression and intelligence are tightly intertwined. Although they cannot be simply equated, the compression rate does capture multiple facets of intelligence—predictive ability, model simplicity, and rapid adaptation. Just as Newton gave "energy" a mathematical definition, compression may provide a quantifiable starting point for understanding intelligence.
For AI practitioners, the practical value of this perspective is: when optimizing language models, the objective can be anchored more directly to the number of bits; when designing compression systems, state-of-the-art prediction models can be confidently embedded into the arithmetic coding framework. The two promote each other and together approach the limits of information theory.
Related articles

Why Git Is Essential in the AI Era: From Version Control to Agent Collaboration
What's the difference between Git and GitHub? How do you manage changes with Issues, branches, and Pull Requests after AI Agents write code? Based on NetworkChuck's Git tutorial, this article walks through the complete flow from git init to conflict resolution and analyzes why version control has become an essential skill in the AI era.

OpenAI 6.2 Astra Hands-On Review Gone Wrong: Impressive Benchmarks, But Developers Say It's Unusable
OpenAI 6.2 Astra leads benchmarks at lower cost, but early testers report frequent stalls in long tasks, unparseable output, and uncontrolled resource management. We examine the gap between real-world experience and official data, and what it means for developer adoption.

GTA VI Coffee System Details Revealed: Hot Coffee Can Be Splashed on People?
Vergecast podcast reveals GTA VI's coffee system: players can brew in apartments, buy from vending machines, order in restaurants, and even splash hot coffee on others. This detailed recreation of daily interactions continues Red Dead Redemption 2's immersive design and sparks discussion on game detail versus gameplay balance.