The Second Law of Thermodynamics and the Limits of AI: The Universe's Perplexity Is Destined to Increase

A tweet grafting thermodynamic entropy increase onto language model perplexity reveals the physical boundaries of AI optimization goals.
Starting from a tweet that analogizes the second law of thermodynamics to machine learning perplexity, the article explores what language models' pursuit of lower perplexity means at the physical level. It introduces perplexity's central role in NLP as an information-theoretic metric, then clarifies that thermodynamic entropy and Shannon entropy, while mathematically connected, are not the same concept—equating them is analogical slippage. Landauer's principle shows information processing necessarily produces entropy: while models lower perplexity, they emit heat into the environment. The article then addresses three dimensions—model capability boundaries, computational cost, and humility about "understanding"—reminding AI practitioners that perplexity cannot decrease indefinitely and that pursuing zero perplexity is physically untenable. Ultimately, the metaphor's value lies in grounding optimization objectives on a more honest physical foundation, and locally organizing order remains the meaning of information processing work.
The Profound Proposition Behind a Single Tweet
Recently, a tweet from Twitter sparked discussion in the tech community: "You can't fight the second law of thermodynamics. The perplexity of the universe is destined to increase."

This sentence cleverly grafts the principle of entropy increase from physics onto the machine learning concept of "perplexity." On the surface it reads as a joke, but it touches on a question worth taking seriously: what does the "perplexity reduction" that language models pursue actually mean at the physical level?
Perplexity: From Information Theory to Language Models
Perplexity is a classic metric in information theory, used to measure a probabilistic model's predictive capability on samples. In natural language processing, a lower perplexity means the model is more "confident" in predicting the next word and has modeled the text better. Models like the GPT series and BERT continually set new perplexity records during training—this is almost the yardstick of progress for the entire field.
But the tweet's author shifts the perspective to the cosmic scale. The second law of thermodynamics tells us that the entropy of an isolated system does not spontaneously decrease. And in information theory, entropy corresponds directly to "uncertainty"—Shannon entropy measures precisely the average amount of information. From this angle, the universe as a whole tends toward greater disorder and greater unpredictability—in machine learning terms, "rising perplexity."
Perplexity is mathematically defined as the geometric mean of the inverse probability the model assigns to a test sequence, equivalent to the exponential form of cross-entropy: $\text{PPL} = \exp(-\frac{1}{N}\sum_{t=1}^N \log p(w_t \mid w_{<t}))$. Intuitively, it represents "on average, among how many equally likely candidate words the model hesitates at each prediction step." For example, a perplexity of 20 means the model's uncertainty is roughly equivalent to choosing uniformly at random among 20 words. In character-level or word-level language models, perplexity is directly tied to compression rate: lower perplexity means higher compression efficiency. It should be noted that perplexity depends heavily on vocabulary size and tokenization method, and values under different vocabularies are not directly comparable; moreover, perplexity only measures how well the model fits the data and does not directly reflect generation quality or factual accuracy.
The Subtle Misalignment Between Two Kinds of "Entropy"
A clarification is needed here. Thermodynamic entropy in physics and Shannon entropy in information theory are deeply connected in mathematical form, but they are not the same thing. Thermodynamic entropy describes the logarithm of the number of microstates, while Shannon entropy describes the uncertainty of a probability distribution. Equating the two directly is a common analogical slippage.
That said, the analogy is not entirely unfounded. Landauer's principle states that erasing one bit of information requires dissipating at least kT ln2 of energy—there is a hard constraint between information processing and thermodynamics. This means that any computation—including training and running large language models—is accompanied by the generation of entropy. While a model lowers perplexity on the training set, it creates more entropy in the environment. This is a debt that cannot be escaped.
Practical Reminders for AI Practitioners
From an engineering perspective, this tweet's jest has several layers worth chewing on.
The Boundaries of Model Capability
A language model's perplexity cannot decrease indefinitely. With limited training data, limited computational resources, and unavoidable noise, a model's modeling of the world will always retain residual uncertainty. Pursuing "zero perplexity" is physically untenable, because it would mean information compressed to the limit without any cost.
Cross-Entropy Loss is the most commonly used objective function in language model training, taking the form $-\sum_t \log p(w_t \mid w_{<t})$, i.e., the negative logarithm of the probability the model assigns to the true next word. Minimizing cross-entropy is mathematically equivalent to minimizing perplexity, since perplexity is the exponential of cross-entropy. Cross-entropy can be decomposed into the entropy of the true data distribution plus the KL divergence between the model distribution and the true distribution: $H(p, q) = H(p) + D_{KL}(p | q)$. Here $H(p)$ is the inherent uncertainty of the data itself, which the model cannot eliminate—it can only minimize the KL divergence term. This explains from an information-theoretic perspective why "zero perplexity" is unreachable: as long as the data itself contains intrinsic randomness, cross-entropy has an unbreakable lower bound.
The Real Cost of Computation
Training and inference of large models consume enormous energy and generate heat—a direct manifestation of the second law of thermodynamics. The industry's pursuit of model efficiency—smaller models, fewer parameters, lower inference costs—is essentially a local game against entropy increase. You can lower the entropy of some subsystem, but at the cost of increasing the entropy of the external environment.
Humility About "Understanding"
If the universe's perplexity is destined to increase, then any model's "understanding" of the world can only be local and approximate. This is not technological pessimism but a clear-eyed recognition of the nature of modeling activity. What a model can compress are the patterns in data, not the world itself.
A Metaphor Worth Keeping
The value of this tweet lies not in its physical rigor but in the interdisciplinary thinking framework it offers. Placing machine learning metrics in the context of thermodynamics prompts a reexamination of things treated as purely engineering problems—perplexity, loss functions, computational efficiency—behind which more fundamental physical constraints are actually at work.
For AI researchers, acknowledging these constraints does not mean giving up on optimization; it means grounding optimization objectives on a more honest foundation. The universe's perplexity may be destined to increase, but humanity can still, locally and within limited scope, organize chaos into usable order. This is precisely the meaning of all information processing work.
Background Notes
Landauer's principle, proposed by IBM physicist Rolf Landauer in 1961, is a bridge between information theory and thermodynamics. It states that at room temperature, erasing one bit of information requires dissipating at least $kT \ln 2$ of energy (about $3 \times 10^{-21}$ joules, where $k$ is the Boltzmann constant and $T$ is the absolute temperature). This principle binds the "logical irreversibility" of information to the "heat dissipation" of physics: any irreversible computational operation must emit heat into the environment. The energy efficiency of modern computing chips is already several orders of magnitude above the Landauer limit, but that limit remains valid as a theoretical lower bound. For large-scale neural network training, Landauer's principle provides a fundamental perspective: reducing the information entropy inside a model (i.e., perplexity) necessarily comes at the cost of increasing environmental entropy—there is no "cost-free intelligence."
Related articles

FBI Emergency Alert: How Autonomous AI Hacking Tools Breach Networks at Machine Speed
FBI emergency alert names open-source AI attack framework Artex, which can turn a plain-language sentence into a full intrusion chain and breach critical industry networks in under three hours. This article breaks down the three-stage attack mechanism, the reality of runaway open-source code, and the FBI's three hardening measures: SIEM intelligence ingestion, cloud least privilege, and legacy asset patching.

Gemini 4 Hydrogen Benchmark Scores Explained: Coding and AI Agent Performance
Full breakdown of Google Gemini 4 Hydrogen benchmarks: SWE-bench 13.5%, Terminal Bench 57%, OS World 77.6%, Automation Bench 76.76%, 1M token context, API from $1.5 per million input tokens, coding and AI agent capabilities analyzed.

FBI Warns About AI Hacking Tool Artex: Systems Breached in Three Hours
FBI warns about AI hacking tool Artex: describe targets in natural language for automated scanning and intrusion, with some attacks achieving full privileged access within three hours. It affected over 2,000 Microsoft 365 organizations and South Korean banks. Copies still circulate after the developer took it down in October; security fundamentals remain key.