Can gzip Be a Language Model? The Deep Logic of Compression as Prediction

A Counterintuitive Question

When we talk about language models, we typically picture neural networks with billions of parameters, massive compute clusters, and complex attention mechanisms. But a seemingly absurd question has been sparking debate in the tech community: can gzip — a general-purpose compression algorithm born in the last century — actually play the role of a language model?

The question comes from Nathan's blog post "Can gzip be a language model?", which earned 263 upvotes and 92 comments on Hacker News. It touches on a profound yet often overlooked principle: compression and prediction are fundamentally the same thing.

Why Compression and Prediction Are Two Sides of the Same Coin

Shannon, the founder of information theory, revealed this long ago: a model that compresses data well must have a good grasp of the data's underlying distribution. The core of compression is eliminating redundancy — and eliminating redundancy requires the ability to predict. If you can accurately predict the next character or word, you can encode it with fewer bits.

In other words, the higher the compression ratio, the deeper the model's "understanding" of the data. The cross-entropy loss that modern large language models optimize during training is, at its core, minimizing the encoded length of text. GPT-series models can be thought of as extraordinarily powerful, lossy (or near-lossless) text compressors. This is precisely why the question arises: if compression is prediction, can a compression algorithm be used in reverse — as a predictor, as a language model?

This equivalence has a rigorous mathematical foundation. Shannon entropy H(X) defines the lower bound on the information content of a random variable — the theoretical limit of lossless compression. The sum of negative log probabilities for each token output by a language model is exactly an estimate of the encoding length for that sequence — which is the origin of the "Perplexity" metric. Lower perplexity means the model fits the probability distribution of the text better, equivalent to encoding that text with fewer bits. Therefore, training a language model to minimize cross-entropy loss and training an optimal compressor to minimize average encoding length are mathematically identical optimization objectives. This perspective is further generalized by Kolmogorov complexity theory: the "true complexity" of a piece of data is defined as the length of the shortest program that can generate it, and intelligent systems can be understood as approximately approaching this theoretical limit.

What the gzip Classification Experiment Reveals

This kind of discussion isn't purely theoretical. Researchers have previously demonstrated that combining gzip with a k-nearest neighbors algorithm for text classification can rival some neural network approaches — which is remarkable. The principle: if two pieces of text belong to the same category, compressing them together yields a higher compression ratio, because they share more patterns and redundant information.

This so-called "Normalized Compression Distance" (NCD) provides a similarity measure that requires no training and no parameters. gzip here plays the role of an implicit "model" — its ability to model data distributions is cleverly converted into classification capability.

The NCD formula is: NCD(x, y) = [C(xy) - min(C(x), C(y))] / max(C(x), C(y)), where C(x) is the compressed length of string x, and C(xy) is the compressed length of their concatenation. The intuition: if x and y are highly similar, the compressor can heavily reuse patterns learned from x when processing y, so the concatenated compressed file won't be much larger than either alone, and NCD approaches 0. If the two are unrelated, NCD approaches 1. This method was formally proposed by Cilibrasi and Vitányi in 2005. Its elegance lies in being completely unsupervised, requiring no domain knowledge — a general-purpose compressor alone can estimate similarity between arbitrary data types (text, DNA sequences, music). A 2023 paper showed that gzip + kNN outperformed some BERT variants on few-shot text classification tasks, sparking widespread discussion — though follow-up research noted that its advantage diminishes significantly on large datasets.

The Gap Between Compressor and Generator

Using gzip as a classifier is already surprising; using it as a generative language model goes a step further. In theory, any compression algorithm corresponds to a probabilistic model: the encoding length a compressor assigns to some data reflects the probability it believes that data has of occurring. From this, we can reverse-engineer a character-level or word-level probability distribution and use it to generate text.

However, a gap exists between theory and reality. The DEFLATE algorithm used by gzip is based on LZ77 sliding windows and Huffman coding. It primarily captures local repetition patterns within a limited window size (typically 32KB) and cannot establish long-range semantic relationships. Even if it could generate text, it would only be reproducing low-order patterns — nowhere near understanding the grammar and semantics of language. This is categorically different from the global dependency modeling enabled by Transformer's self-attention mechanism.

The DEFLATE algorithm consists of two stages. The first is LZ77: it maintains a sliding window (up to 32KB in standard gzip) and matches the byte sequence at the current position against content already seen in the window. When a repetition is found, it replaces the original bytes with an "offset + length" pointer, eliminating local redundancy. The second stage is Huffman coding: it builds a variable-length prefix code for the symbol set output by the first stage, representing high-frequency symbols with shorter codes to further reduce output size. These two mechanisms define the upper limit of gzip's "language modeling capability": it can only use historical information within the 32KB window and cannot perceive paragraph-level or document-level semantic structure. In contrast, Transformer's self-attention mechanism can arbitrarily establish dependencies between any tokens within the entire context window, encoding global statistical patterns from training data into billions of parameters — a fundamental difference in modeling capability.

The Real Value of This Thought Experiment

Discussing whether gzip can become a language model was never about whether it could truly replace GPT. The point is that it helps us see the underlying mathematical nature of language models more clearly. When a technology becomes mystified and over-hyped, returning to the most fundamental information-theoretic perspective often brings much-needed clarity.

For practitioners, this perspective carries several practical implications: first, model quality can be evaluated through the lens of compression ratio; second, in resource-constrained scenarios, simple compression methods may provide surprisingly strong baselines; third, it reminds us that a large part of intelligence may simply be efficient compression of the world's patterns. The lively discussion on Hacker News also reflects that engineers maintain a deep interest in this kind of first-principles reasoning.

Conclusion

gzip is not, and will never become, a true language model in any meaningful sense — it lacks the ability to understand semantics, handle long contexts, or generalize. But the value of the proposition "compression is intelligence" lies in bringing neural networks down from the pedestal and back onto solid theoretical ground. The next time we marvel at the capabilities of a large model, it's worth remembering: at its core, it is still an extraordinarily sophisticated compression machine that has learned to predict the next token.