[KongchangAI]
· 5 min read· 2,504 words

Superintelligence in Just Two Years? AI Research Automation Will Ignite the Singularity

Superintelligence in Just Two Years? AI Research Automation Will Ignite the Singularity

AI is accelerating its own R&D; verifiability could make AI research automation arrive by 2028, bringing faster and smarter automated researchers.

Based on the METR report and Anthropic internal data, the author points out that AI is already significantly accelerating AI R&D, with coding agent usage surging within months. Explosive progress in verifiable domains such as mathematics shows that AI research itself relies even more on verifiable empirical progress than mathematics does, making automation more feasible. The author predicts a median year of 2028; automated researchers have the triple advantages of being faster, smarter, and more numerous, which will greatly compress algorithmic progress cycles. The future may lead to a "software singularity" or a "sparks" scenario, each with roughly equal probability. The lack of continual learning, the data bottleneck, and experimental compute limits all have verifiable solution paths.

From Ten Years to Two: One AI Researcher's Drastic Change in Perspective

If you had asked me earlier this year how far away superintelligence is, I would have said at least ten years. But over the past three months, my view has shifted dramatically. After repeated debates with people both on and off the podcast, and after observing AI's progress in verifiable domains (such as math and programming), I now believe: we may be only two years—or even a few months—away from AI reaching top human level in AI research.

This judgment didn't come out of nowhere. It's based on a trend that is already happening: AI has begun to significantly accelerate AI's own R&D. According to a three-month study by the independent evaluation organization METR inside Anthropic, Anthropic's AI progress on well-defined tasks is twice as fast as it was a year ago, and most of that acceleration "is likely attributable to Anthropic's extensive use of AI for R&D." OpenAI has also disclosed that among its internal researchers, the top 10% most frequent users of coding agents saw their daily consumption soar from about $1,000 in early summer to over $7,000 by the end of August. Anthropic reports that Claude currently dominates 26% of the company's AI R&D work—up from less than 1% in February.

These data points all point to the same conclusion: AI is becoming the core productivity engine of AI research. Once AI can fully automate AI research, algorithmic progress will no longer be limited by the number and speed of human researchers.

Breakthroughs in Mathematics: The Power of Verifiable Tasks

The key evidence that changed my mind comes from AI's explosion in mathematics. Just two years ago, large language models couldn't reliably do four-digit addition; in 2024, they could barely solve high school math competition problems; in 2025, they were already winning gold medals at the International Mathematical Olympiad; and this year, they have begun solving open problems that were previously unresolved. In the fully verifiable domain of mathematics, AI is producing important results at an astonishing rate—problems that hundreds of top mathematicians spent decades trying to solve are being solved in batches by AI as "side quests."

Of course, current AI is better at verification-intensive mathematical work such as finding counterexamples rather than proposing new theories. But the key point is: AI research itself relies even more on "verifiable progress" than mathematics does. The ultimate goal of mathematical research is to expand humanity's understanding of mathematical concepts; proving conjectures is merely a means. AI research, by contrast, has the opposite priority—labs only care whether a new algorithmic recipe can empirically make training more efficient and models smarter. The progress of deep learning over the past fifteen years has required almost no new theory.

This is exactly where AI excels. Take the Transformer architecture: core components like attention mechanisms and residual networks had long existed in the literature, but no one knew how to combine them correctly. Inventing the Transformer required extremely strong research taste, and even more so, repeated trial and error across many small experiments. And "continuously and intelligently matching dozens of combinations of ideas" is precisely the superpower AI has demonstrated. You could say that the Transformer—the biggest breakthrough in machine learning since AlexNet—is exactly the kind of thing that Claude-generation models might invent if asked to "try hard to make attention mechanisms work better."

AI research automation will dramatically accelerate algorithmic progress

Verifiable tasks refer to problems whose correctness can be judged automatically and objectively—for example, step-by-step checking of mathematical proofs, pass rates on programming problem test cases, or win/loss determination in games. Such tasks provide AI with dense and cheap feedback signals, enabling rapid iteration through reinforcement learning or large-scale trial and error. By contrast, open-ended creative writing or strategic decision-making lacks immediate verification, so AI progresses much more slowly there. The article's core argument is built on this distinction: AI research (especially algorithmic experimentation) is naturally verifiable, and therefore will be automated sooner than expected.

Timeline: AI Research Automation by 2028

If you ask me for a median year, I believe AI research automation will arrive around 2028, provided AI labs push full steam ahead and are not slowed down by governments or their own decisions. This judgment is based on a simple pattern: in every domain where answers can be checked and AI can be given thousands of practice rounds—whether competitive programming or well-defined open mathematical problems—AI goes from "worse than most people" to "better than the very best humans" in just a few years.

Some will object: the core ability of top AI researchers—such as betting on the right research direction years in advance—cannot be directly verified. But mathematics has already proven that models can generalize far beyond their training data. OpenAI's models were obviously not trained by solving other Millennium Prize problems; they were trained on large numbers of simpler, more tractable math problems, yet generalized from them to solve problems humans had never cracked. The same applies to AI research: thousands of rounds of practice on verifiable tasks, plus each generation of models being smarter than the last, will make models perform excellently even in research steps that cannot be directly checked.

Of course, this milestone could also be delayed. I estimate at least a 25% probability that it arrives after 2032. But conversely, things could also go much faster—some people with good track records predicting over the past few years believe AI could beat the very best human AI researchers as early as next year. If you plug METR's "AI progress speed doubles every year" data into takeoff models, former guest Ajay Akochar's inference is: "This naively means we will reach extreme superintelligence within a year." My median is not 2027 only because I doubt the accuracy of such direct modeling. But we should be prepared for this milestone to arrive next year.

AI agents processing massive tasks in parallel

Faster, Smarter, More: The Three Advantages of Automated Researchers

Once AI reaches top human level in AI research, algorithmic progress will accelerate dramatically for three reasons: faster, smarter, more.

Today's AI already works about 10 times faster than humans, and can run day and night, weekends included. That means each automated researcher completes about 50 times the weekly workload of a human researcher. Of course, this doesn't mean AI progress will immediately accelerate 50-fold—experimental compute remains a bottleneck, and diminishing returns are inevitable. But even with extremely steep diminishing returns, this alone could multiply the speed of algorithmic progress several times over.

More critical is "smarter." In a typical frontier lab, about 80% of algorithmic progress is driven by the top 50 researchers. Zuckerberg's willingness to spend hundreds of millions of dollars recruiting top talent is no accident. In an automated company, every engineer and researcher operates at the level of the best people in today's top human labs. Add "more"—AI research contains a great deal of modular work that can be independently verified, such as improving pretraining data filtering, designing synthetic RL environments, writing better attention kernels, and so on. In these areas, deploying large numbers of continuously improving agents can often significantly accelerate progress.

Recursive self-improvement (RSI) refers to an AI system's ability to improve its own code or architecture, thereby producing a stronger next generation of AI and creating a positive feedback loop. This concept is the core mechanism in discussions of "intelligence explosion." The key tension in the article is that the acceleration effect of RSI competes with the drag of diminishing returns in AI research itself. If AI's ability to improve itself grows faster than problems become harder, a "software singularity" may occur; otherwise, a "sparks" scenario emerges. Understanding RSI helps grasp why the author assigns roughly equal probability to the two futures rather than confidently betting on one.

Software Singularity or Sparks? Two Futures

Automated researchers will accelerate AI progress, but will that speed persist? Here there are two competing trends: on one hand, recursive self-improvement (RSI) makes automated researchers faster, smarter, and more numerous; on the other hand, AI research becomes increasingly difficult as easy ideas are eaten up by the swarm. AI companies need to quadruple R&D compute and double headcount every year just to maintain the same pace of progress.

If AI becomes smarter faster than ideas become harder, we will see a "software singularity": without adding compute, the AI swarm rapidly iterates out smarter versions within months, until it hits the physical limits of current chips. Conversely, there is the "sparks" scenario: one year accomplishes three to six years of AI progress at current speeds, then slows down waiting for more compute. I think both are equally likely.

But even the "sparks" scenario is crazy enough. In the past four years, we went from GPT-3.5—a "toy" that could write doggerel and autocomplete simple Python functions—to models that can build complete video games from scratch and produce a large number of mathematical contributions worthy of a Fields Medal. Going from "preschooler" to "genius intern" in four years, and then taking another equally large step, you get an intelligence far beyond humans in slightly verifiable domains: better than Nvidia at designing the next generation of AI accelerators, more profitable than Citadel and Jane Street given equal market access, and—if remote experimentation is possible—better than all of humanity at finding room-temperature superconductors.

Verifiability Is the Underlying Fabric of AI Research

Why am I confident that AI research is "verifiable enough"? AI R&D roughly divides into three parts: pretraining, post-training, and inference efficiency, each with strong verifiable proxies.

The core metric for pretraining is loss on a held-out test set, which is extremely easy to verify. You can use RL to train a model to minimize the compute needed to train a small model to a fixed loss, then deploy that model to optimize the perplexity of a larger model. Research taste—such as predicting whether a small-scale improvement will hold at large scale—is also trainable: have the model predict whether an improvement that works at 100 million parameters will still hold at 10 billion parameters.

Post-training has three verifiable proxies: improving post-training algorithms so that RL consumes less compute/data/time on new environments; improving the model's ability to learn on the fly in environments it has never been trained on; and directly measuring AI R&D capability. Anthropic's internal CoBench contains about 500 real problems that researchers solved this year, and each benchmark is also an RL environment, letting models practice redoing the progress frontier humans made over the past three to six months.

Inference efficiency is even more highly verifiable: GPU utilization, KV cache size per token, total FLOPs per forward pass, latency—all can be measured precisely.

Inference efficiency metrics are highly verifiable

The Strongest Rebuttal: Continual Learning and the Data Bottleneck

The strongest objection is this: models lack continual learning ability, and every session is like "the first day on the job," unable to accumulate weeks or months of work experience. That was my main reason for doubting AI research automation last December. But I have now partly changed my view. First, AI companies may have solved continual learning by then—by extending in-context learning so that a single session can effectively last for months. Second, you may not need a continual learner to solve continual learning: online learning itself is an ML research problem with clean verifiable proxies. Current models can already solve Millennium Prize problems while in a "forgetful" state—like keeping up with Bolt while doing somersaults. What if we just let them run normally?

The data bottleneck is also solvable. Research by me and Princeton student Jerry Han found that over the past six years of pretraining, data improvements contributed more than three times as much algorithmic progress as model improvements. But data improvement itself is a typical AI research task: figuring out which filtering or synthetic data techniques reduce loss on held-out corpora. DeepSeekAd's paper also points out that changes to the RL algorithm itself had minimal impact, while "systematic improvements in the scale, diversity, and verifiability of synthetic data and environments explain almost all observed gains." The production of data is being automated.

The experimental compute bottleneck is also less severe than imagined. Frontier labs more than quadruple compute every year. If the trend continues, by the end of 2027, OpenAI could give almost 10,000 agents—roughly the number involved in solving a Millennium Prize-scale problem—each enough compute to run a GPT-3-scale training run once a day. Today's labs "waste" compute on experiments because researcher time is far scarcer than compute. When the bottleneck flips, automated researchers can run large numbers of small-scale ablations, read every rollout record, and spend subjective decades designing the most informative aggregate experiments. They can also discover better ways to learn from small experiments—just as the GPT-4 paper used a model with 10,000 times less compute to predict final loss, automated companies can search for more scaling phenomena like muP.

Continual learning refers to a model's ability to keep learning from new data or tasks without forgetting old knowledge. Current large language models start each session from fixed weights and cannot accumulate months of experience like human researchers, which is regarded as one of the fundamental obstacles to automating AI research. The author proposes two mitigation paths: first, extend context windows so that a single session can effectively span months; second, treat "achieving online learning" itself as a verifiable ML research problem and let AI automatically search for solutions. This reflects the article's consistent logic: as long as a problem can be defined with clear evaluation metrics, AI has a chance to conquer it through trial and error.

Conclusion

Judging by the pace of progress in verifiable domains, AI research automation is no longer a science fiction ten years away. 2028 is a reasonable median, and 2027 is entirely possible. Once we get there, whether "software singularity" or "sparks," we will face an intelligence far beyond humans along verifiable dimensions. This does not necessarily mean immediate utopia or doom, but it does mean the pace of progress we are familiar with will be radically compressed. It's time to take this timeline seriously.

One-line Summary

AI is accelerating its own R&D; verifiability could make AI research automation arrive by 2028, bringing faster and smarter automated researchers.

Paragraph Summary

Based on the METR report and Anthropic internal data, the author points out that AI is already significantly accelerating AI R&D, with coding agent usage surging within months. Explosive progress in verifiable domains such as mathematics shows that AI research itself relies even more on verifiable empirical progress than mathematics does, making automation more feasible. The author predicts a median year of 2028; automated researchers have the triple advantages of being faster, smarter, and more numerous, which will greatly compress algorithmic progress cycles. The future may lead to a "software singularity" or a "sparks" scenario, each with roughly equal probability. The lack of continual learning, the data bottleneck, and experimental compute limits all have verifiable solution paths.

#AI research automation#superintelligence#recursive self-improvement#METR#verifiable tasks
Share:

Related articles