Design Principles of AI Mathematical Solving Systems: A Complete Guide to LEAN Formal Proofs

Recent breakthroughs in AI for mathematics have attracted widespread attention, particularly systems capable of generating formal proofs. A developer shared on Reddit their understanding of how these systems work, expressing a desire to build their own version to explore problems in high-dimensional geometry. The discussion revealed the core design philosophy behind today's AI mathematical solving systems.

Core Workflow: Generate-Verify-Iterate

Based on the developer's observations, these mathematical solving systems typically employ the following architecture:

First, the system asks a model (usually DeepMind's Aster or similar) to generate mathematical statements in LEAN format. LEAN is an open-source interactive theorem prover initially developed in 2013 by Leonardo de Moura at Microsoft Research. Built on Dependent Type Theory, it encodes mathematical theorems and proofs into formalized expressions that can be rigorously verified by computers. The widely used Lean 4 version is not only a proof assistant but also a general-purpose programming language. The Mathlib library built around LEAN has formalized over 150,000 mathematical theorems spanning algebra, analysis, topology, and many other branches, providing AI systems with a rich foundation of verified mathematical knowledge. The generated statements are then submitted to the LEAN compiler for verification — its type checker acts as a "mathematical referee," where any logically flawed proof step causes compilation failure, thus eliminating the implicit assumption errors or logical leaps that human mathematicians might make.

Aster is an AI system introduced by DeepMind in 2025 specifically designed for mathematical reasoning and formal proof generation. Before Aster, the field had already seen several milestone developments: Meta's HyperTree Proof Search in 2022 first demonstrated neural network-guided theorem proof search capabilities; DeepMind's AlphaProof solved silver medal-level problems at the International Mathematical Olympiad in 2024; and that same year, the AlphaGeometry system achieved gold medal-level performance on geometry problems. Aster represents the latest advancement in this direction, capable of handling more abstract and longer chains of mathematical reasoning.

The key innovation lies in the feedback mechanism: the system uses the LEAN compilation result (success or failure) to determine whether the generated statements are valid. If compilation passes, these statements are added as verified "facts" to serve as the foundation for subsequent reasoning. This process iterates continuously until the complete proof compiles successfully in LEAN.

This design essentially combines intuitive mathematical reasoning (provided by the AI model) with rigorous formal verification (provided by LEAN), leveraging both the generative capabilities of language models and ensuring mathematical rigor.

The Challenge Beyond Context Windows

The developer raised an important question: how do you handle extremely long proofs? Some AI-generated mathematical papers span hundreds of pages, far exceeding the context window limits of any model.

The context window refers to the maximum text length a large language model can process in a single inference, measured in tokens. Even the most advanced current models (such as Claude's 200K tokens, GPT-4 Turbo's 128K tokens, and Gemini's 1 million tokens) face fundamental limitations when confronted with hundreds of pages of mathematical proofs. This isn't just a length issue — as context grows, the model's attention to information in the middle sections decays, a phenomenon known as "Lost in the Middle." The deeper challenge is that logical dependencies in mathematical proofs often span very long distances: a lemma used on page 200 might have been proven on page 15. This means simply expanding the context window doesn't truly solve the problem; external structured knowledge management mechanisms are essential.

This implies the system must employ some kind of "chunk-and-build" strategy. Proofs aren't generated all at once but assembled incrementally. The system likely uses a layered architecture:

Fact Management System

Maintain a knowledge base of verified propositions, ensuring each proof step is built on a reliable mathematical foundation. This knowledge base needs to support efficient retrieval and logical relationship tracking, so when the system needs a lemma with specific properties, it can quickly locate relevant verified results.

Subproblem Decomposition

Break large problems into manageable pieces, with each part processed independently. This strategy mirrors the natural structure of mathematical research — proofs of complex theorems typically depend on a series of intermediate lemmas, each of which is an independent, verifiable mathematical result.

Incremental Construction

After each piece is independently verified, it's gradually combined as a component of the larger proof. In LEAN, this composition is natural: a theorem verified in one file can be directly referenced in other files, and the compiler automatically checks the correctness of dependencies.

State Tracking

Record the current proof progress and available known facts to ensure clear reasoning paths. This is typically implemented through a dependency graph — a directed acyclic graph data structure where each node represents a mathematical proposition and edges represent logical dependencies, enabling the system to precisely track which lemmas are used by which theorems.

This design resembles how human mathematicians work: first prove lemmas, then build more complex theorems on top of them. The key challenge is how to "meaningfully combine smaller ideas into larger ideas" — precisely the core challenge the developer faces.

Practical Advice and Feasibility Analysis

For developers looking to build their own version, here are several key considerations:

Getting Started

You don't need enormous hardware resources for meaningful exploration. Start with simple geometry problems using existing open-source models (such as smaller models accessed via API) paired with a LEAN environment. The key is designing good data structures and retrieval mechanisms for your "fact library." Lean 4 installation and configuration are quite mature — you can quickly set up a development environment using elan (Lean's version manager) and lake (the build system), while Mathlib as a pre-built math library provides a wealth of ready-made definitions and lemmas as starting points.

Combination Strategies

Try the following approaches for combining ideas:

1. Use dependency graphs to track logical relationships between propositions — this not only helps the system understand proof structure but also enables precise retrieval when specific types of lemmas are needed
2. Implement simple "proof search" algorithms, similar to theorem provers — common strategies include breadth-first search to systematically explore all possible proof paths, best-first search using heuristic functions to prioritize the most promising directions, and Monte Carlo Tree Search (MCTS), the method used by systems like AlphaProof, which models proof search as a sequential decision problem similar to Go, evaluating the potential of different proof strategies through simulation and backpropagation
3. Have the model first generate a proof outline, then progressively refine each step — this "coarse-to-fine" strategy closely aligns with how mathematicians actually work, first establishing the macro framework of a proof, then filling in the details of each step

Potential Innovation Points

• Combine Retrieval-Augmented Generation (RAG) to extract relevant lemmas from mathematical literature — RAG is a technical architecture that integrates external knowledge retrieval with language model generation. In mathematical proof scenarios, the system can retrieve the most relevant known lemmas and proof techniques from vast mathematical literature databases (such as hundreds of thousands of papers on arXiv, or formalized theorem libraries in Mathlib) for the current proof goal. A key challenge lies in semantic retrieval of mathematical content — traditional text similarity metrics often fail to capture deep structural similarities between mathematical objects, requiring specially trained mathematical embedding models for effective retrieval
• Use reinforcement learning to optimize proof search strategies — in a reinforcement learning framework, the "state" is the current proof progress, the "action" is choosing which tactic or lemma to apply next, and the "reward" comes from LEAN compiler feedback. This approach allows the system to learn efficient search strategies through millions of proof attempts, potentially developing proof search heuristics that surpass human intuition and finding non-obvious proof paths that human mathematicians might overlook
• Develop smarter subgoal selection mechanisms — this involves choosing the most valuable intermediate lemma to prove first among many possibilities, considering both its contribution to the final goal and its own provability

This is by no means a "fool's errand." While cutting-edge systems do require massive computational resources, the core ideas can be validated at smaller scales. Many breakthrough ideas have started with someone's "scrappy version" experiments.

Looking Ahead: The Democratization of Formal Proofs

AI mathematical solving systems represent an exciting direction: combining the rigor of formal verification with AI's creativity. As these technologies mature, we may see:

• More domain experts using these tools to explore their own research problems — the barrier to formal proofs is lowering, and future systems may allow researchers to describe conjectures in natural language, with AI automatically translating them into formal expressions and attempting proofs
• Interactive proof assistants introduced in mathematics education — students can receive real-time feedback on their proof attempts in environments like LEAN, understanding the strict requirements of logical reasoning
• New avenues for solving interdisciplinary problems (such as the high-dimensional geometry the developer is interested in) — high-dimensional geometry problems typically involve spatial structures that are difficult for humans to grasp intuitively, and AI systems can systematically explore properties in high-dimensional spaces, discovering mathematical relationships beyond human intuition

The key takeaway is that these systems aren't black-box magic but carefully engineered systems. Their core — the cycle of generation, verification, and fact accumulation — can be understood and reproduced. This cycle essentially simulates the scientific method itself: formulate hypotheses, verify experimentally, accumulate knowledge, formulate new hypotheses. For developers wanting to contribute to this field, now is an excellent time to explore. Formal mathematics is at a turning point from academic niche to widespread application, and AI is accelerating this transition.