Three AI Researchers Debate: How Far Are We from Recursive Self-Improvement?

Recursive self-improvement (RSI) and intelligence explosion are among the most contested topics in AI today. In a wide-ranging conversation, three researchers from relatively open labs — Zyphra CTO Barron Millich, Thinking Machines Chief Scientist (and former OpenAI co-founder and core RLHF contributor) John Schulman, and Base10 model training lead Charlie O'Neill — engaged in a candid, technically dense debate on these questions. None of them are pure accelerationists or outright skeptics. Instead, they approached the "intelligence explosion" narrative from first principles — training mechanics, data bottlenecks, and generalization — to unpack the real technical constraints behind it.

Why Might There Still Be No Superintelligence in Ten Years?

The opening question was pointed: assuming that ten years from now the world has not been transformed by countless superintelligences, what are the most likely technical reasons (excluding external shocks like war or regulation)?

Barron's core concern was a continuation of Moravec's Paradox — AI performs exceptionally well on the benchmarks and environments we design for it, yet some persistent sim-to-real gap always seems to block the emergence of true generalization. He added that if meta-learning generalization proves extremely difficult, compounded by continual learning remaining unsolved, that would be his default scenario.

John's perspective was more grounded in practitioner experience: humans still outperform models in many areas. Every time a new model is released, people exclaim "this is AGI" — but a month later it feels dumber. This cycle of amazement followed by disappointment may keep repeating, and it's hard to predict how many more iterations remain. The reason there's no explosive growth yet is that even if models can write far more code than humans, research and engineering still get bottlenecked elsewhere, preventing a 100x boost in human productivity.

Charlie approached it from the angle of a global optimum for learners. He drew an analogy to Moore's Law: that beautiful straight line was sustained for so long because it was underpinned by countless discrete technological breakthroughs. LLMs are similar — when the pretraining scaling law hit diminishing returns, RL stepped in and extended the curve. The question is whether the current approach of training LLMs with RL environments is capable of discovering the next discontinuous breakthrough needed to keep that line going upward.

---

Moravec's Paradox, proposed by roboticist Hans Moravec in the 1980s, observes that high-level cognitive tasks humans find difficult (chess, mathematical reasoning) are relatively easy for AI, while low-level sensorimotor tasks humans perform effortlessly (recognizing a face, picking up an egg) are extremely hard for AI. Barron extends this to the sim-to-real gap: models excel in standardized benchmarks (simulation) but exhibit systematic capability degradation when faced with unstructured variables in the real world.

Meta-learning (also called "learning to learn") refers to training paradigms where models learn not just to perform specific tasks, but to rapidly adapt to entirely new task types — essentially learning how to learn more efficiently. Continual learning studies how to enable models to keep acquiring new tasks and data without forgetting existing capabilities. Both are technical prerequisites for RSI: without meta-learning new objectives or continuously accumulating knowledge, self-improvement has no foundation.

---

The Key Bottleneck for RSI: Can AI Define Its Own Goals?

The debate gradually converged on a central proposition: whether RSI can be self-driven depends on whether AI can generalize to "learning its own objectives."

Charlie distinguished between two categories of research. The first is automated research where objectives are clearly defined — such as reducing pretraining loss or maximizing RL environment scores. AI can bring enormous acceleration here. He offered a compelling example: if AI had been thinking about the Kaplan scaling laws back then, it might have immediately noticed that "they used intermediate checkpoints without considering annealing, so the conclusions are wrong" — a single observation that could have saved a year or two of detours. Similarly, insights like muP and how learning rates should scale with model width are "low-hanging fruit" that could have been discovered much earlier with more careful thought. He predicted roughly a 10x speedup on the task of "maximizing current objectives."

But the second category is open-ended science requiring paradigm shifts, where the objectives themselves cannot be clearly specified — and AI can't specify them either. John pointed out that the entire field depends heavily on generalization, and when generalization will occur is extremely hard to predict. The most important advances tend to be those we had no prior reason to expect: generalizing from naive next-token prediction to deep understanding, from verifiable tasks to unverifiable ones. None of these are guaranteed.

One important consensus the three reached: "thinking" cannot conjure new bits out of thin air. As Charlie put it, thinking can only update your posterior probability based on information already acquired — you cannot obtain genuinely new information from pure thought alone. This means even an automated researcher given a century to contemplate each experiment cannot derive correct new objectives from scratch.

Why Has Distillation Prevented Centralization Among Model Providers?

On the topic of industry structure, an interesting question arose: why hasn't there been massive centralization among model providers?

John's answer was that distillation is the primary force countering centralization. Whatever can be learned through RL is essentially a small number of bits and is easily distilled — once you have trajectories demonstrating certain behavior, replication is straightforward. Barron added a practical observation: Chinese labs are very likely using data from routing/proxy services that allow Chinese users to bypass restrictions and access US frontier models (primarily for coding). These services collect and sell data, providing a perfect prompt distribution for distillation.

This led to an intriguing prediction: Charlie observed that despite Anthropic's advantages in distillation and logit distillation, certain open-source models objectively outperform their closed counterparts. He speculated this suggests frontier labs may no longer hold much of an edge in RL environments, and real-world deployment data may matter more than the environments themselves. John offered a slightly different hypothesis: environments can be constructed along axes of "difficulty" and "authenticity." Distillation easily matches teacher model performance on benchmark-gaming distributions, but struggles to capture capabilities needed in genuine, multi-turn interaction scenarios — where large models generalize better from narrow tasks to real-world tasks.

---

Knowledge distillation, systematized by Hinton et al. in 2015, uses a large "teacher model's" outputs (soft labels/logits) to train a smaller "student model," enabling the latter to approximate the teacher's capability distribution with fewer parameters. Compared to training directly on human-annotated data, distillation transfers the teacher's encoding of uncertainty — for example, the teacher believing an answer is 70% correct and 20% likely to be another answer, which is informationally richer than hard correct/incorrect labels.

Logit distillation is a variant that directly aligns the probability distributions over vocabulary between teacher and student, rather than only aligning final answers. John's key insight: capabilities acquired through RL are essentially "strategies compressed into a small number of bits," and once their behavioral trajectories can be observed, they can be efficiently copied via distillation. This explains why frontier capabilities rapidly diffuse to the open-source community — the moat isn't in what the model knows, but in the ability to generalize across genuine multi-turn scenarios, which is harder to capture through distillation.

---

The Technical Ceiling of Continual Learning: Plasticity and Catastrophic Forgetting

The most information-dense portion of the conversation was about why continual learning is so difficult.

Barron divided tasks into two categories: cumulative tasks (like RSI, where each discovery is a "line drawn in sand" that can be preserved — attention, MoE, GRPO and other advances continuously stack in the training pipeline) and non-stationary tasks (like a legal firm assistant, where interpersonal dynamics and implicit rules constantly shift and can't simply be accumulated). He joked: "It's so unfortunate that RSI happens to be easier than almost every other task."

Charlie broke down in detail how current methods fail: when you try to make continuous micro-updates to a single model (say, continuously updating it for a specific law firm), almost every approach collapses — SFT causes catastrophic forgetting and degradation of general capabilities, on-policy distillation can push that horizon back a bit but ultimately fails too, and RL is good at injecting capabilities but not knowledge. He and Barron agreed: this is both a technical and partly a capacity problem, but fundamentally technical — because if you put the same data into mid-training and train a fresh same-size model, the result is better. This is precisely why labs today continuously train new base models rather than updating old ones indefinitely: plasticity loss and catastrophic forgetting form hard constraints.

---

Catastrophic forgetting is a fundamental property of neural networks: when the network updates weights on new tasks, the parameter representations for old tasks get overwritten, causing sharp capability degradation. This differs fundamentally from how the human brain consolidates memories via the hippocampus. Various mitigation approaches have been proposed — Elastic Weight Consolidation (EWC), Replay Buffers, etc. — but none have fundamentally solved the problem at large model scale.

Plasticity loss is a separate dimension: as a model receives more and more fine-tuning updates, its weights gradually "solidify," and its responsiveness to new information continually declines. Even if the loss function keeps decreasing, the model's ability to truly "learn new things" is eroding. These two constraints together explain why continuously updating a single model systematically underperforms training a new model on the same data from scratch — the latter has full plasticity, while the former is over-constrained by historical weights. This is the fundamental technical reason why labs release entirely new base models every few months rather than maintaining a single model indefinitely.

---

Why Has RL Succeeded Beyond Expectations?

Faced with the critique that "RL only learns one bit per episode and learns almost nothing when success rates are low," why has scaling RL proven so effective?

Barron offered two explanations. First, mid-training is severely underrated — much of what appears to be "RL success" actually comes from extremely high-quality mid-training data (pre-training-style training on synthetic reasoning data), which brings the model to roughly 80% of the final RL checkpoint before RL even begins. RL only needs minimal policy fine-tuning from there. Second, signal-to-noise ratio: RL's objective function discards all other bits and retains only the single high-signal bit of "how to get the answer right," so it doesn't get drowned in noise. This is why RL is so surprisingly efficient in steps compared to SFT.

Charlie added the generalization dimension: RL hasn't produced cross-domain "reasoning generalization" (training only on math won't make you the best programmer), but it has produced "temporal span generalization" — models learned to use more tokens and persist longer on tasks. Research shows the duration a model can work continuously is doubling every three months, which is clear evidence of generalization.

---

Mid-training (also called Continual Pre-Training) is the phase between large-scale pretraining and task-specific fine-tuning, typically run on high-quality domain or synthetic data in a pre-training-like manner. Barron emphasizes it is severely underrated: the success of models like DeepSeek-R1 owes much to mid-training on large volumes of synthetic reasoning chain data, rather than pure RL policy optimization. This observation challenges the popular narrative that "RL is the primary source of reasoning capability."

GRPO (Group Relative Policy Optimization) is an RL algorithm variant proposed by the DeepSeek team. It estimates baselines by comparing relative rewards across multiple sampled outputs within a group, avoiding the overhead of training a separate value network as in PPO, and is computationally more suited for large model training. The "stacking of attention, MoE, GRPO and other advances in the training pipeline" that Charlie and Barron reference is exactly RSI manifesting in "cumulative tasks" — each generation of training methods stands on the shoulders of the previous one, forming a limited but real self-improvement loop.

---

The Three Experts' Timeline Predictions

In a rapid-fire round, the three offered predictions on several key milestones:

• A plug-and-play general remote white-collar worker (capable of computer use, learning, and executing complex projects seamlessly for a month): ~1–3 years if organizations are allowed to adapt for AI; ~3 years for fully general capability.
• AI researcher achieving 10x productivity: Barron and Charlie said ~2 years; John gave 5–10 years — the divergence mainly reflects different interpretations of "fully general."
• Surpassing top human experts across all cognitive work performable on a computer (i.e., ASI): Estimates concentrated in the 3–10 year range. John noted that AI research, having received enormous attention and involving code and math where models excel, may not actually be the hardest domain. The real long tail lies in data-scarce areas requiring in-context learning — like becoming a superhuman engineer at TSMC.

Notably, several researchers suggested that "automated AI research" may already be "ASI-complete" — because the world contains so many things that are hard to master even with external memory systems and longer context windows. The value of this conversation lies not in providing definitive answers, but in reducing the grand narrative of "intelligence explosion" to a series of specific, debatable, and verifiable technical propositions.