Science as a Recursively Self-Improving System: Why AI Won't Produce an "Intelligence Explosion"

A discussion about AI recursive self-improvement (RSI) has recently attracted attention. The author puts forward a thought-provoking argument: if you want to truly understand AI's recursive self-improvement, the best reference frame isn't the "intelligence explosion" from science fiction films, but rather a recursively self-improving system that has existed in the real world for centuries — science itself.

This perspective treats science as an intelligent system, even as an "agent" with goals and resources. By observing the evolutionary patterns of the scientific system over hundreds of years, the author reaches a conclusion that diverges from mainstream "Singularity" theory: AI's recursive self-improvement is indeed happening, but it will not lead to an exponential intelligence explosion.

Science: An Obvious Recursively Self-Improving System

The author first argues that science meets all the hallmarks of recursive self-improvement. Scientific discoveries feed back to accelerate scientific progress itself, creating multiple positive feedback loops:

• New technologies unlock better experimental tools: Scientific discoveries yield new technologies, which in turn help build more sophisticated experimental equipment — the primary driver of progress in almost every field.
• New concepts enable new breakthroughs: Advances in theory and ideas make more problems solvable.
• Economic output funds research: Science increases society's economic output, channeling more resources back into science.
• Computing power and tool iteration: Better chips and networking technology produce faster tools, creating a closed technology-tool loop.

Because of these feedback loops, several "input" metrics for science show exponential growth: the number of researchers roughly doubles every 15 years, global R&D spending roughly doubles every 13 years, papers and patents continue to expand, and the computing power devoted to science roughly doubles every 2 years.

Exponential Input Growth, Yet Linear Output

Here lies the most counterintuitive and crucial argument in the piece. Given that inputs are growing exponentially, is the pace of scientific progress also growing exponentially? The answer is no.

The author notes that since the Industrial Revolution, the rate of scientific impact output has remained roughly constant — in other words, scientific progress is linear. He uses three 50-year periods for comparison:

1850–1900

Evolution, germ theory and antiseptic surgery, thermodynamics, electromagnetic field equations, the periodic table, the pharmaceutical industry, electricity, the telegraph and telephone, the internal combustion engine, skyscrapers, mechanized agriculture…

1900–1950

Special and general relativity, quantum mechanics, nuclear fission and atomic energy, antibiotics, genetic theory, electronic computers, information theory, synthetic polymers and plastics, the transistor, aviation…

1950–2000

DNA, genetic engineering, integrated circuits, microprocessors and personal computing, the internet, crewed spaceflight, the moon landing, satellite communications and GPS, the Standard Model of particle physics…

Each 50-year list of breakthroughs is equally impressive. The author also cites a particularly compelling example: human life expectancy has increased at an almost perfectly linear rate of roughly 3 months per year since 1840. This pattern of "straight-line progress" is especially evident in fields like biology, medicine, and agriculture, where impact is easier to quantify.

Why Exponential Input Growth Yields Only Linear Output

The author wrote about this phenomenon over a decade ago and notes that subsequent research has continued to confirm it. He mentions two representative works: Nielsen and Collison's Science Is Getting Less Bang for Its Buck, and the economics paper Are Ideas Getting Harder to Find? The author even suggests the former was inspired by a conversation he had with Nielsen.

The core reason: research always prioritizes the highest-impact, most tractable problems first, and each subsequent problem is either harder or lower-impact — and exponentially so.

A vivid example is information theory — that foundational paper was written by a single author and wasn't particularly difficult to write, yet almost no computer science paper since has come close to matching its impact. The low-hanging fruit gets picked first; the remaining fruit hangs ever higher.

This explains why the scientific system requires exponentially increasing resource inputs just to produce linear gains in impact. Over time, achieving equivalent progress becomes exponentially more difficult.

The economics paper Are Ideas Getting Harder to Find? (Bloom et al., 2020, published in the American Economic Review) quantifies the long-term decline in research productivity through empirical analysis of multiple industries including semiconductors, agricultural breeding, and pharmaceuticals. In semiconductors, for instance, the number of researchers required to sustain Moore's Law doubles roughly every 13 years, meaning each researcher's "idea output" decays at the same rate. Nielsen and Collison's research similarly finds that the impact of top journal papers (measured by subsequent citations and real-world application) has continuously declined relative to inputs. These empirical results independently corroborate the article's central argument from both an economics and scientometrics perspective: the expansion of the knowledge frontier follows diminishing returns, not exponential acceleration. This pattern stems from the structural characteristics of the problem space itself and does not change as the scale of inputs grows.

Implications for AI Recursive Self-Improvement

The author's position is quite clear: he fully believes AI's recursive self-improvement is already happening and will accelerate in the future. But he does not believe this will lead to an "intelligence explosion."

The reason is precisely the historical lesson provided by the scientific system. If even science — a recursively self-improving system that has operated for centuries, commands vast resources, and features multiple positive feedback loops — can only produce linear progress, then AI's self-improvement likely follows a similar pattern: the more you invest, the greater the marginal difficulty, rather than uncontrolled, limitless acceleration.

In other words, the existence of recursive self-improvement does not equal an exponential explosion in capability. Feedback loops are offset by the fundamental constraint that problems keep getting harder. The author believes that the Singularity-style intelligence explosion scenario contradicts both his understanding of intelligence and everything he knows about recursively self-improving systems in the real world.

This analysis offers a rare moment of calm reflection amid today's heated debates about AGI and superintelligence. It reminds us that observing a real, quantifiable, and historically rich self-improving system is often more persuasive than purely theoretical reasoning.

The concept of the "Intelligence Explosion" was first proposed by mathematician I.J. Good in 1965: once machine intelligence surpasses human intelligence, it can design even smarter machines, triggering a self-accelerating recursive loop that ultimately produces a superintelligence far beyond human comprehension in an extremely short time. This idea was later widely popularized by Ray Kurzweil's "Technological Singularity" theory and became the theoretical foundation for many catastrophic risk assumptions in AI safety. The core premise of Singularity theory is that improvements in intelligence lead to exponential growth in capability, which in turn further accelerates improvements in intelligence, creating an uncontrolled positive feedback loop. The author's argument directly challenges this premise — even with multiple positive feedback loops, real-world complex systems may not produce a true exponential capability explosion in any meaningful sense.