Regression to the Mean: How LLMs Are Quietly Killing Innovation and Diversity

When AI Becomes an Engine of Averages

In statistics, regression to the mean describes a simple yet profound phenomenon: extreme values tend to drift back toward the average in subsequent observations. The concept was first identified by British statistician Francis Galton in the 19th century during his research on heredity — he observed that the children of unusually tall parents tended to "regress" toward the population average height. Notably, Galton's discovery carries a deeper statistical implication: when two variables are imperfectly correlated, the appearance of extreme observed values is partly driven by random variation, so subsequent observations naturally "shrink" back toward the mean. Nobel laureate Daniel Kahneman later invoked this principle in Thinking, Fast and Slow to expose a systematic bias in human causal reasoning — we tend to misread regression to the mean as the effect of some intervention. Since then, the pattern has shown up everywhere: financial markets, medical research, athletic performance. When we project this concept onto large language models (LLMs), a troubling picture emerges — these powerful AI systems are, at their core, engines that pull everything toward the average.

This idea sparked heated debate on Hacker News: while LLMs are driving a productivity revolution, they may simultaneously be quietly strangling the truly "new" — the unexpected, the unconventional, the genuinely original idea or expression.

The Mathematical Nature of LLMs Is Regression to the Mean

From a technical standpoint, this claim is no exaggeration. The core mechanism of an LLM is predicting the next most probable token, meaning every output is essentially a search for the highest-probability path through the training data distribution. Highest probability almost always means most common, most typical, most "average."

This mechanism is grounded in the self-attention mechanism and Softmax probability distribution of the Transformer architecture. Introduced by Google in the 2017 paper Attention Is All You Need, the Transformer's key innovation is self-attention — allowing the model to dynamically weigh the importance of all other tokens in a sequence when generating each new token. During generation, the model computes a Softmax probability distribution over the vocabulary (typically 32,000 to 128,000 tokens), producing a probability vector that sums to 1. The "temperature" parameter controls sampling randomness: as temperature approaches 0, the model outputs the highest-probability token (most conservative); as temperature rises, more randomness is introduced (more creative but less stable). Mathematically, the temperature parameter T controls the "sharpness" of the distribution by scaling logits before Softmax (logits/T) — which means so-called "creative mode" is essentially artificially injected random noise, not genuine original thinking.

In other words, LLMs are optimized to generate content that "looks most reasonable" — and the flip side of "most reasonable" is "least surprising." True innovation is almost always born in low-probability territory: the rare, marginal, even seemingly "wrong" combinations that seldom appear in training data. When a system is designed to consistently choose high-probability outputs, it is structurally at odds with breakthrough creativity.

The Paradox of Innovation: Rarity Being Systematically Diluted

Why "the New" Is So Fragile

Genuine novelty has always been a scarce resource in human civilization. A disruptive scientific hypothesis, an unprecedented artistic style, an expression no one has ever uttered before — these are precious precisely because they deviate from existing patterns. In cognitive science, the theory of "remote associative thinking" was proposed by psychologist Sarnoff Mednick in 1962, who developed the famous Remote Associates Test (RAT) to quantify creativity. Neuroscience research shows that creative thinking is closely linked to the brain's Default Mode Network (DMN) — a network most active during unstructured thought. A 2019 study published in PNAS found that highly creative individuals exhibit stronger asynchronous coupling between the DMN and the executive control network, enabling concepts with great semantic distance to be activated simultaneously and connected. This mechanism is fundamentally different from how LLMs work: the "remoteness" of human creative thinking stems from the parallel activation properties of biological neural networks, while LLMs' autoregressive generation always moves along high-probability paths through training data — precisely the blind spot that high-probability prediction models struggle to reach.

The problem with LLMs is that they flatten this deviation. As more and more writing, coding, design, and research is completed with LLM assistance, we are effectively letting an averaging engine intervene at every stage of human creation. This gradual homogenization of thought won't arrive as a single catastrophe — it will accumulate through countless small, nearly imperceptible "regressions to the mean."

The Danger of Feedback Loops: Model Collapse

The deeper threat lies in the circular contamination of data. As AI-generated content floods the internet, that content becomes training data for the next generation of models, creating a self-reinforcing feedback loop:

First-generation models learn from human-created content, producing outputs that trend toward the average
These averaged outputs become training material for second-generation models
Second-generation models further narrow the diversity of their outputs
The cycle repeats, and the "tails" of the distribution gradually disappear

Academics call this model collapse. In 2023, a team led by Ilia Shumailov at Oxford published a study in Nature that systematically described this effect through an information-theoretic framework: when a model is iteratively trained on its own generated data, it is effectively performing lossy compression on the original data distribution — each iteration discarding low-frequency information from the distribution's tails. From a KL-divergence perspective, the divergence between the nth-generation model's output distribution and the original human data distribution increases monotonically with each iteration, while distributional entropy decreases monotonically. Experimental data shows that after 5–10 rounds of iterative training, model coverage of low-frequency vocabulary and long-tail knowledge drops by over 40%, while surface-level fluency shows virtually no perceptible degradation — which is precisely what makes this dangerous: the decay is silent and difficult to detect with conventional quality evaluation methods. Rare, long-tail knowledge (specialized terminology, niche culture, fringe perspectives) disappears first, and the model eventually converges toward a kind of "cultural mean state." This closely mirrors entropy reduction in information theory — a system in a closed loop spontaneously moving toward an ordered but impoverished state. The concern about regression to the mean resonates deeply with these research findings.

"The Quiet Death": The Invisible Cost to Innovation

The Hidden Trade-off Between Efficiency and Innovation

The phrase "the quiet death" is strikingly evocative. It suggests that the demise of innovation won't happen dramatically, but will proceed in silence beneath the halo of efficiency.

The productivity gains LLMs deliver are real, visible, and quantifiable — emails written faster, code produced more easily, proposals drafted more efficiently. But what they suppress is an invisible opportunity cost: ideas that could have been born but never were; the deep exploration abandoned because "the AI's answer was good enough."

It's nearly impossible to calculate the loss of a breakthrough that never happened — which is why this cost is so easy to overlook. When everyone can effortlessly access "average excellence," the motivation to pursue "exceptional distinctiveness" may be quietly eroded. Clayton Christensen, in his 1997 book The Innovator's Dilemma, revealed the core paradox of disruptive innovation: truly industry-reshaping breakthroughs almost never emerge from the "optimized" path of mainstream players — they enter from peripheral markets serving non-mainstream users with underperforming products. Nearly every paradigm-level breakthrough in history — from quantum mechanics to the internet, from Impressionism to rock and roll — was an "outlier" at the tail of the distribution when it first appeared. This logic bears a deep structural resemblance to LLMs' averaging tendency: LLMs are optimized to serve the "average needs" of the "average user" — which is precisely the least fertile ground for disruptive innovation to take root.

The Debate: The Other Side of Tool Neutrality

This warning has drawn clearly divided responses. Some argue that LLMs are indeed fueling intellectual homogenization; others counter that human culture has always been full of imitation and repetition, that genuine originality was always the province of the few, and that LLMs are simply making this pre-existing dynamic more visible.

A middle-ground perspective is also worth considering: LLMs are tools, and the degree of regression to the mean depends on how they're used. Treating an LLM as a "standard answer generator" will indeed reinforce mediocrity. But positioning it as a "sparring partner for ideas" or "a guide for exploring the edges" might actually push humans to break out of conventional thinking. The tool itself is neutral — what matters is the intent and awareness of the person using it.

How to Resist AI's Regression to the Mean: Protecting Innovative Outliers

Reframing the Limits of LLM Output

Facing this challenge, the most important step is a shift in awareness. When using an LLM, we need to be clear-eyed: it gives you the safest answer, not the best one. Truly valuable thinking often requires actively deviating from the AI's default suggestions.

In innovation research, the "outlier" corresponds precisely to the mechanism by which disruptive innovation is born. Protecting innovative outliers is essentially protecting those low-frequency but high-value neural connections in the human cognitive network — connections that are systematically down-weighted in an LLM's probability distribution, yet are the critical nodes of civilizational progress.

A few practical strategies are worth considering:

Use LLMs for routine, repetitive tasks; reserve creative thinking for human leadership
Actively challenge the model with "counter-intuitive" questions, deliberately exploring the tails of the probability distribution
Maintain a critical eye toward AI outputs — don't readily accept the "good enough" default answer
At key innovation junctures, preserve space for purely human collaboration and unstructured brainstorming

Redefining AI's Role in the Creative Process

LLMs' nature as averaging engines is determined by their mathematical foundations — it cannot be eliminated through simple technical patches. The real solution lies in how we define AI's role in the creative process.

If we let AI take the lead in "generating," homogenization becomes inevitable. But if we position AI as a tool for amplifying human judgment — keeping final creative decision-making firmly in our own hands — then the death of "the new" might be delayed, or even reversed.

In an era where AI is penetrating every stage of creation, protecting those precious "outliers" in human thought may be our most important — and most easily overlooked — responsibility.