Why Is DeepMind Falling Behind on Math Benchmarks? The Clash Between Specialized Systems and General-Purpose Models

A Surprising Phenomenon

When it comes to AI breakthroughs in mathematics, many people's first thought is Google DeepMind. Over the past few years, this lab has practically defined the field of "AI doing math": from AlphaGeometry, which cracks geometry olympiad problems, to AlphaProof, which performs formal theorem proving, to AlphaEvolve, which discovers new solutions through evolutionary algorithms—DeepMind has the deepest research foundation in mathematical reasoning and search.

Yet recently, a Reddit user raised an intriguing question: why is it OpenAI, not DeepMind, leading the current math benchmark leaderboards?

"If you had asked me a year or two ago which lab would dominate hard math benchmarks, I would have almost certainly said DeepMind. The fact that it isn't leading in what has historically been its strongest area really surprises me."

Behind this question lies a deeper divergence among frontier AI labs in technical approaches, product strategy, and evaluation standards.

The Clash Between Specialized Systems and General-Purpose Models

To understand this phenomenon, we first need to distinguish between two fundamentally different types of "AI doing math."

DeepMind's "Specialized Weapons" Approach

AlphaGeometry, AlphaProof, and AlphaEvolve are essentially specialized systems custom-built for specific mathematical tasks. They typically combine large language models with symbolic search, formal verification engines (like Lean), or evolutionary algorithms—carefully designed hybrid architectures.

Specifically, AlphaGeometry is a geometry theorem-proving system released by DeepMind in early 2024. It combines a language model trained on synthetic data with a symbolic reasoning engine (DDAR), capable of solving plane geometry problems at near-IMO gold medalist level without relying on human demonstrations. AlphaProof is built on the formal mathematical language Lean 4, capable of translating natural language mathematical propositions into formal statements and then searching for rigorous proofs through reinforcement learning-driven search processes. Lean is an interactive theorem prover that requires every reasoning step to be machine-verified within a type theory framework, so proofs produced by AlphaProof possess absolute mathematical rigor—fundamentally different from natural language reasoning generated by large language models. AlphaEvolve transforms mathematical problems into program search problems, maintaining a population of candidate solutions, using large language models to generate mutation operations for new candidates, and then filtering excellent individuals through automated fitness evaluation. It has already discovered new results on the cap set problem in combinatorial mathematics that surpass the best-known human solutions.

These systems deliver stunning performance on International Mathematical Olympiad (IMO)-level problems—AlphaProof even achieved silver medal level.

But their defining characteristic is: deep but narrow. These systems are highly optimized for specific domains like geometric proofs and formal theorems, not general-purpose chat models you can casually invoke. They typically don't participate in direct competition on general benchmark leaderboards, because the two are fundamentally evaluating different things.

OpenAI's "General-Purpose Model" Approach

The popular math benchmarks today largely evaluate general-purpose large language models' end-to-end performance on mathematical problems. Current mainstream math benchmarks include GSM8K (elementary math word problems), MATH (high school competition mathematics covering algebra, geometry, number theory, etc.), AIME (American Invitational Mathematics Examination problems), and newer ones like FrontierMath. These tests typically present problems in natural language, requiring models to provide final numerical answers or proof steps, with scoring usually based on exact matching. As model capabilities improve, earlier benchmarks like GSM8K have essentially been saturated, and the industry is shifting toward more challenging test sets. But the common feature of these benchmarks is evaluating the full pipeline from reading problems to outputting answers—fundamentally different from the formal theorem proving paradigm that requires every step to be machine-verified.

OpenAI has made rapid progress in this area recently, with their blog even publishing contributions their models made on ten difficult math problems. These achievements come from overall improvements in general model capabilities, not specialized math systems.

In other words, DeepMind is far ahead on the "specialized track," but on the "general model leaderboard" track, Gemini's optimization direction isn't entirely aligned with OpenAI's. This is one of the core reasons for the ranking differences on leaderboards.

Different Optimization Objectives

Beyond architectural approaches, the two labs' strategic priorities also differ significantly.

Gemini Leans Toward Productization

Google has deeply integrated Gemini into its massive product matrix spanning Search, Workspace, Android, and more. This means Gemini's optimization objectives are broad practical capabilities and deployment at scale—multimodal understanding, long context processing, tool use, cost control, and more—all dimensions that need to be balanced.

The pressure of deployment at scale shouldn't be underestimated. AI Overview in Google Search handles billions of queries daily. Workspace assistants need to understand complex contexts across documents, emails, and spreadsheets. Android system-level integration requires models to run efficiently on edge devices. This means the model must simultaneously meet standards across inference latency, computational cost, multilingual support, safety alignment, and more. An optimization that gains a few extra percentage points on math benchmarks might not be worthwhile at the product level if it comes at the cost of doubled inference time or significantly increased deployment costs. This also explains why Gemini's engineering investment focuses more on inference efficiency optimization (like the Flash version) and long context capability (million-token windows).

Under such an objective function, extreme mathematical reasoning may not be the top priority.

OpenAI Is More Willing to Showcase the Frontier

By contrast, OpenAI has been more aggressive over the past year in showcasing their models' ceiling on difficult reasoning tasks, using math, code, and other hardcore capabilities as flagship talking points. This "flexing" strategy naturally makes it easier to gain exposure and leading positions on math benchmark leaderboards.

So DeepMind "not leading" doesn't equal "lacking capability"—it's very likely they simply haven't concentrated resources on competing on these public leaderboards.

The Limitations of Benchmarks Themselves

There's another angle worth being cautious about: to what extent can these math benchmarks represent genuine mathematical ability?

Public leaderboards have several inherent issues:

• Data contamination risk: Popular test sets may have entered training data, causing artificially inflated scores. Data contamination is a serious challenge facing AI evaluation today—since large language model training data typically comes from internet crawls, and popular benchmark problems and answers are widely available across forums, blogs, and academic websites, models may have "seen" test problems during pretraining. Research shows that even partial leakage (such as seeing problems but not standard answers) can significantly boost model scores on benchmarks.
• Inconsistent evaluation criteria: Whether tools are allowed, whether formal verification is used, and the difficulty distribution of problems all significantly affect rankings. Differences in evaluation settings across labs—such as whether multiple sampling with best-of-k (pass@k) is allowed, whether chain-of-thought prompting is used, whether code execution tools are enabled—can cause the same model to show drastically different scores in different reports, making direct cross-lab comparisons extremely complex.
• Differing incentives to showcase: Not all labs will run their strongest internal systems on public leaderboards.

Systems like DeepMind's AlphaProof pursue verifiable rigorous proofs, not high scores on multiple-choice-style benchmarks. Although both are called "math," the dimensions being measured are completely different. Using general benchmarks to judge a lab focused on formal proofs may inherently be a mismatch.

How to View This "Misaligned Competition"

Taken together, "DeepMind not leading math benchmarks" is more of an evaluation perspective issue than a decline in capabilities.

This can be understood on three levels:

1. Technical depth: On the most hardcore mathematical proofs and novel solution discovery, DeepMind's specialized systems remain in a leading position.
2. General capability: For consumer-facing general-purpose large models, Gemini and the GPT series each have different emphases—OpenAI is more willing to showcase upper limits on math leaderboards.
3. Strategic trade-offs: Google puts more effort into product integration and scaling rather than short-term rankings on public benchmarks.

For practitioners following AI progress, the lesson from this case is: don't be misled by a single leaderboard. Whether a lab is "leading" depends on what ruler you use to measure. Mathematical ability spans multiple dimensions—olympiad problem-solving, formal proofs, open-ended research—and no single benchmark can fully capture the complete picture.

What's truly worth anticipating may be when DeepMind gradually integrates specialized capabilities like AlphaProof and AlphaEvolve into the Gemini general-purpose model—that's when comparisons might get closer to the essence of the competition.

Key Takeaways