StochBench: A Deep Dive into the First Lean Formalization Benchmark for Stochastic Processes

A New Frontier for Formal Mathematics

Current mainstream benchmarks for formal theorem proving, such as IMO (International Mathematical Olympiad) and Putnam competition problem sets, are certainly challenging but primarily focus on competition mathematics, making them poor representatives of the practical needs in specific academic disciplines. Formal theorem proving refers to the process of using rigorous computer programming languages to express and verify mathematical theorems — unlike traditional pen-and-paper proofs, formal proofs require every reasoning step to be checked by a computer verifier, eliminating the possibility of human error. A recent research paper from arXiv introduces StochBench — a Lean 4 benchmark specifically designed for the field of stochastic processes, filling a critical gap in formal proof for applied mathematics.

Lean 4 is a next-generation interactive theorem prover and functional programming language developed by Microsoft Research. It can serve both as a general-purpose programming language and as a tool for constructing and verifying mathematical proofs. Compared to its predecessor, Lean 4 offers significant improvements in performance and usability, featuring a more modern compiler architecture with support for metaprogramming and custom tactics, enabling users to flexibly extend proof automation capabilities.

This benchmark contains 450 graduate-level stochastic process problems, each accompanied by its natural language source text, spanning multiple levels of abstraction from foundational to advanced. The field of stochastic processes has long been underrepresented in Mathlib (Lean's core mathematics library), and StochBench was created precisely to bridge this gap. Mathlib is one of the largest single formalized mathematics libraries in existence, collaboratively maintained by hundreds of contributors worldwide. As of now, it contains over one million lines of formalized code, covering broad mathematical branches including algebra, analysis, topology, and number theory. However, Mathlib's coverage is uneven — pure mathematics branches like algebra and analysis are relatively well-developed, while applied mathematics areas such as stochastic processes, partial differential equations, and numerical analysis remain significantly under-formalized. This imbalance directly impacts AI systems' ability to perform automated proofs in these fields, as the lack of foundational lemmas and definitions means AI must build reasoning chains from much lower levels.

Coverage and Technical Depth of StochBench

StochBench's problem coverage reflects the complete knowledge graph of stochastic process theory. Stochastic processes form the core theoretical framework in probability theory for studying random phenomena that evolve over time, with mathematical foundations deeply rooted in measure theory — the modern axiomatic probability system established by Lebesgue, Kolmogorov, and others. Under the measure-theoretic framework, a probability space is defined as a triple (Ω, F, P), where Ω is the sample space, F is a σ-algebra (a collection of events), and P is a probability measure. Conditional expectation is no longer a simple division of conditional probabilities but rather an almost-everywhere unique measurable function defined through the Radon-Nikodym theorem. This highly abstract mathematical foundation makes formalizing stochastic processes particularly difficult, as every seemingly intuitive concept requires rigorous measure-theoretic support behind it.

StochBench specifically covers the following core topics:

• Markov Processes: Finite and countable Markov chains, continuous-time Markov processes. The defining characteristic of Markov processes is "memorylessness" — the probability distribution of future states depends only on the current state, regardless of past history. In a formal environment, defining the Markov property requires precisely expressing the dependence of conditional probability on σ-algebras, involving the concept of filtration — a family of increasing σ-algebras used to model the gradual revelation of information. Finite-state Markov chains can be compactly represented through transition matrices, but generalizations to countable states and continuous time require handling convergence of infinite-dimensional matrices and semigroup theory, which demands substantial infrastructure in Lean.
• Random Walk Theory: Classical random walks and their variants
• Martingale Theory: Martingale processes, stopping time theory. The martingale is one of the most elegant and profound concepts in probability theory, originating from the study of gambling strategies and later developing into a core tool of modern stochastic analysis. Intuitively, a martingale describes a "fair game" — given knowledge of past information, the expected future value equals the current value. A stopping time is a special random variable representing the first time a certain event occurs, with the key property that the decision to "stop or not" can only be based on current and past information. The Optional Stopping Theorem connects these two concepts and holds a foundational position in financial derivative pricing — the mathematical foundation of the Black-Scholes option pricing formula is precisely martingale measure theory.
• Renewal Processes: Renewal theory and its applications
• Queueing Theory: Modeling of various queueing systems
• Brownian Motion and Stochastic Calculus: Brownian motion, Itô integrals, and other advanced topics. Brownian motion (also known as the Wiener process) is the cornerstone of continuous-time stochastic processes, characterized by three key properties: independent increments, normally distributed increments, and continuous sample paths. Despite having continuous paths, it is almost everywhere non-differentiable — this counterintuitive property means classical calculus tools cannot be directly applied. Itô calculus was developed precisely to address this problem, founded by Japanese mathematician Kiyosi Itô in 1944. Unlike the classical Riemann-Stieltjes integral, the Itô integral requires special attention to the choice of evaluation points in the integral approximation (left endpoints rather than midpoints), as different choices lead to different integral results (compare with the Stratonovich integral). Itô's lemma — the chain rule of stochastic calculus — is a ubiquitous core tool in financial mathematics, physics, and engineering.
• Convergence Theory: Weak convergence, Poisson processes

These topics are not only central to courses on probability theory and stochastic processes but also form the mathematical foundation for applied fields such as financial engineering, operations research, and communication theory. By formalizing this content as Lean 4 code, StochBench provides a more practically relevant standard for evaluating AI capabilities in specialized mathematical domains.

A Litmus Test for AI Proof Capability: What Does a 34.9% Success Rate Mean?

The research team systematically tested StochBench using an Opus 4.8-based agent, achieving a success rate of 34.9% (157/450) under a 15-minute time limit per problem. The proving agent employs an architectural pattern where a large language model interacts with a formal verifier: the LLM generates sequences of proof tactics, while the Lean compiler verifies each reasoning step in real time. When a tactic fails, the agent adjusts its strategy based on error messages, forming a closed loop of "generate-verify-revise." The advantage of this approach lies in combining the LLM's pattern recognition and intuitive guessing abilities with the absolute rigor of the formal verifier. The 15-minute time limit means the agent must find the correct proof path within a limited search budget, placing high demands on the efficiency of tactic selection.

This data conveys two key messages.

On one hand, the success rate confirms that StochBench is genuinely challenging. Even an advanced large language model paired with specialized proving strategies could only solve roughly one-third of the problems. This stands in stark contrast to competition benchmarks like the IMO, where some problems have been nearly fully solved.

On the other hand, the 34.9% success rate also demonstrates substantial progress in AI's formal mathematical proving capabilities. Stochastic processes involve extensive abstract concepts, limit theory, and measure-theoretic foundations. This performance indicates that AI systems have developed preliminary abilities to handle specialized mathematical problems, extending well beyond elementary mathematics or competition tricks.

Implications for AI Mathematical Reasoning Research

Traditional formal theorem proving benchmarks tend to emphasize discrete mathematics and elementary algebra, while the concepts of continuity, randomness, and measure theory introduced by StochBench bring entirely new dimensions of challenge for AI systems. Competition mathematics problems from IMO and Putnam are typically "self-contained" — they require a limited scope of knowledge, with the key lying in clever combinations and reasoning techniques. Theorem proving in applied mathematics is fundamentally different: it often depends on vast theoretical frameworks, where proving a single theorem may require invoking dozens of prerequisite lemmas spanning multiple mathematical branches. For example, proving a large deviation principle in stochastic processes may simultaneously involve topology (compactness arguments), functional analysis (dual spaces), and measure theory (weak convergence). This kind of cross-domain knowledge integration is one of the core bottlenecks currently facing AI systems.

For instance, proving properties of Brownian motion requires understanding the topological structure of continuous function spaces; proving the martingale convergence theorem requires manipulating advanced probabilistic concepts such as conditional expectation and almost sure convergence. The reasoning depth required for these problems far exceeds that of typical competition mathematics.

This type of domain-specific benchmarking is crucial for advancing AI's practical value in scientific research and engineering applications. Rather than merely chasing high scores on competition mathematics, building reliable formalization capabilities in actual academic disciplines is the true core direction for AI-assisted mathematical research.

Future Outlook: The Trend Toward Vertical Specialization in Formal Math Benchmarks

The release of StochBench marks a new phase of vertical specialization in formal mathematics benchmarking. As more domain-specific benchmarks emerge, the following development trends can be anticipated:

1. More Precise Capability Assessment: Designing specialized tests for different mathematical branches rather than applying one-size-fits-all general benchmarks
2. Deep Integration of Domain Knowledge: Encouraging AI models to learn domain-specific proof patterns and intuitions
3. Practical Formalization Tools: Driving continued expansion of formalized mathematics libraries like Mathlib into applied mathematics domains

For researchers in probability theory, statistics, and related applied fields, StochBench is not merely an evaluation tool — it has the potential to evolve into an auxiliary platform for verifying proof correctness in both teaching and research. When AI can reliably handle these specialized problems, mathematicians will gain an efficient formal verification partner.

Key Takeaways