Study Group Guide for Bach's "Learning Theory from First Principles": Building Machine Learning Theory from the Ground Up

A Collaborative Experiment in the Mathematical Foundations of Machine Learning

Recently, a compelling community initiative appeared on Reddit: a machine learning enthusiast launched a reading group for Francis Bach's Learning Theory from First Principles (MIT Press, 2024). The organizer candidly admitted that while the book is excellent, its dense content and intricate proofs make solo study extremely challenging. The goal is to bring together a group of like-minded learners to tackle the mathematical difficulties through weekly discussions.

One major advantage of this book is that it's completely free — the author has published the PDF on his personal website, eliminating any financial barrier. This also lowers the cost of joining the study group, making it accessible to anyone interested in machine learning theory. For the many practitioners and students flooding into the AI field today, this approach of returning to mathematical first principles represents a valuable path that's often overlooked amid the prevailing "just import a library" culture.

What Does Francis Bach's Book Cover?

Learning Theory from First Principles is a major work by Francis Bach. Bach is a senior researcher at the French National Institute for Research in Digital Science and Technology (INRIA) and a leading authority in machine learning optimization and kernel methods. INRIA (Institut National de Recherche en Informatique et en Automatique) is France's most important computer science and applied mathematics research institution, occupying a central position in the AI research landscape across Europe and globally. Bach has long led INRIA's SIERRA team, focusing on convex optimization, kernel methods, and sparse modeling. His publications have been cited tens of thousands of times, and he has served as program chair for ICML (International Conference on Machine Learning). This book can be seen as the systematic theoretical distillation of over two decades of research experience, with the core objective of building a theoretical framework for supervised learning systematically from mathematical foundations.

Overview of the Book's Content

According to the organizer, the book follows a progressive knowledge path:

• Starting point: It begins with least squares and empirical risk minimization (ERM), which form the cornerstone of statistical learning. The fundamental idea behind ERM is that since we cannot directly minimize the unknown true risk (i.e., the expected loss of a model over the entire data distribution), we instead minimize the empirical risk computed on a finite set of training samples as a proxy. The theoretical roots of this framework trace back to the pioneering work of Vapnik and Chervonenkis in the 1970s, who proved that under certain conditions (such as finite VC dimension of the hypothesis space), the minimizer of empirical risk can converge to the minimizer of the true risk at a controllable rate. Understanding why ERM works and under what conditions it fails is the starting point for understanding the entire edifice of machine learning theory — from overfitting, regularization, to model complexity control, nearly all key concepts revolve around this framework;
• Optimization theory: Discussing how to solve the optimization problems underlying learning;
• Local averaging methods: Including classic nonparametric methods such as k-nearest neighbors;
• Kernel methods: This is Bach's own area of expertise and a critical bridge connecting linear models to nonlinear modeling. The core idea of kernel methods is to implicitly map data into a high-dimensional (or even infinite-dimensional) feature space via a kernel function, thereby using linear methods in that space to solve nonlinear problems in the original space. The mathematical foundation is Reproducing Kernel Hilbert Space (RKHS) theory. Kernel methods, represented by support vector machines (SVM), were the dominant paradigm in machine learning during the 2000s. Although the rise of deep learning has pushed kernel methods to the sidelines in terms of applications, their theoretical value has only increased — recent research has revealed deep connections between wide neural networks and kernel methods (such as Neural Tangent Kernel / NTK theory), making the theoretical framework of kernel methods an important key to understanding deep learning;
• Model selection: How to navigate the bias-variance tradeoff;
• Neural networks: Extending the theoretical perspective to deep learning models.

Advanced Topics: Overparameterization and PAC-Bayes

In later chapters, the book also addresses cutting-edge topics in current academia, including overparameterized models and the PAC-Bayes framework.

The overparameterization puzzle is one of the most perplexing theoretical problems of the deep learning era. Classical statistical learning theory follows the bias-variance tradeoff principle: overly complex models (too many parameters) lead to overfitting and degraded generalization performance. However, modern deep neural networks often have billions of parameters — far exceeding the number of training samples — yet still perform remarkably well on test data. Several explanatory attempts have emerged in recent years: implicit regularization theory suggests that gradient descent algorithms inherently tend to find "simple" solutions; the double descent phenomenon shows that after model complexity surpasses the interpolation threshold, test error may decrease again rather than continuing to rise; additionally, the geometric structure of the loss landscape and the special inductive biases of neural networks are also considered key factors. This area remains one of the most active frontiers in machine learning theory research, with no unified explanatory framework yet.

The PAC-Bayes framework provides a Bayesian-perspective tool for analyzing generalization bounds. PAC-Bayes is a fusion of PAC (Probably Approximately Correct) learning theory and Bayesian methods, first developed by McAllester (1999) and Catoni, among others. Its core idea is: rather than analyzing the generalization performance of a single hypothesis, it analyzes the expected generalization performance of a probability distribution (posterior distribution) over the hypothesis space. The general form of PAC-Bayes bounds shows that the upper bound on generalization error is proportional to the KL divergence between the posterior and prior distributions — intuitively, if the data hasn't forced your beliefs to deviate too far from the prior, the model is more likely to generalize well. This framework has attracted renewed attention in the deep learning era because it can yield tighter generalization bounds than traditional VC dimension or Rademacher complexity, and is particularly well-suited for analyzing stochastic learning algorithms (such as stochastic gradient descent).

These topics demonstrate that the book doesn't stop at classical statistical learning theory but actively engages with modern deep learning practice.

Study Group Design: Rotating Presentations and Collective Discussion

The organizer has a clear plan for the study group, reflecting a commitment to high-quality discussions.

Organizational Format

• Weekly online calls of approximately one hour: One member presents the week's key conclusions, followed by a collective walkthrough of unclear details;
• Discord community: Providing a space for questions and discussion between meetings.

This "rotating presentations + collective discussion" model is essentially a practice of active learning. The Feynman Technique, named after Nobel Prize-winning physicist Richard Feynman, holds a core principle: if you can't explain a concept in simple, clear language to someone else, you don't truly understand it. This method has solid theoretical support in cognitive science — research on the "generation effect" shows that actively generating information (such as explaining or teaching) creates stronger memory encoding and deeper conceptual understanding than passively receiving information. Moreover, when explaining to others, learners are forced to expose gaps in their understanding — this "metacognitive monitoring" process is a key mechanism for efficient learning. In math-intensive learning scenarios, this method is especially effective — whether each step of a complex proof is truly understood becomes fully exposed when attempting to explain it to others. Compared to passive reading, this approach can significantly improve the absorption of dense mathematical content.

Prerequisites and Background Requirements

The organizer explicitly listed recommended background knowledge: linear algebra, probability theory, and comfort with reading mathematical proofs. One notable detail: they specifically emphasized that "you don't need a theory background, just the willingness to sit with the details." This statement is quite apt — the barrier to learning theory isn't whether you've taken specific courses, but whether you have the patience and determination to dig deep. In fact, linear algebra and probability theory are the two pillars of machine learning theory: the former provides the language for handling high-dimensional data and model parameters (matrix decomposition, eigenvalue analysis, projections, etc.), while the latter is the foundation for understanding statistical estimation, generalization error, and uncertainty quantification. With these two foundations plus the ability to read formal proofs, you're ready to enter the world of learning theory.

Why Is This Kind of ML Theory Study Group Worth Paying Attention To?

In today's rapidly evolving AI landscape, the vast majority of learning resources focus on engineering practice and tooling, while systematic study of the underlying mathematical theory is often marginalized. However, as the book's title suggests — "from first principles" — truly understanding why machine learning works and when it fails requires a solid grasp of these mathematical foundations.

"First principles" thinking originates from physics and refers to reasoning from the most fundamental axioms and assumptions, rather than relying on analogies or rules of thumb. In the machine learning context, this means not merely knowing empirical rules like "more data improves performance" or "regularization prevents overfitting," but understanding why these methods work — what is the precise relationship between sample complexity and hypothesis space complexity? What geometric constraints does regularization introduce into the optimization objective? Which probability inequalities does the proof of generalization bounds rely on? This deep understanding empowers practitioners to construct solutions from the ground up when facing new problems, rather than merely applying existing tools.

The Value of Small Learning Communities

The organizer plans to keep the group at five to ten people, reasoning that it should be "small enough so that discussions actually work." This consideration is quite pragmatic. Overly large study groups often devolve into formality with only a handful of active participants; a moderately sized group with highly committed members is more likely to sustain learning momentum and foster deep intellectual exchange.

From a learning science perspective, this size choice also has its merits. The concept of "Dunbar's number" and research on small group dynamics in education both suggest that five to eight people is the optimal size for in-depth technical discussions: diverse enough to bring different perspectives and complementary skills (some members may have stronger mathematical foundations, others more programming experience), yet small enough to ensure everyone has ample opportunity to speak and participate. In mathematical theory study, different members may approach the same proof from different angles — some through geometric intuition, others through algebraic manipulation — and this collision of diverse perspectives often catalyzes understanding deeper than any individual could achieve alone.

Open and Free Knowledge Sharing

It bears repeating that the book's PDF is freely available from the author. This reflects the growing spirit of open sharing in academia — leading scholars willing to share the fruits of years of research with learners worldwide at no cost. This practice continues the longstanding open tradition in the machine learning field: from the arXiv preprint culture, to Andrew Ng's free courses on Coursera, to Goodfellow, Bengio, and Courville's freely available Deep Learning textbook, this field has consistently maintained a high level of knowledge accessibility. Bach's choice to publish through MIT Press while simultaneously providing a free PDF balances the rigor of academic publishing with the inclusiveness of knowledge dissemination. Combined with a free Discord community, this study group project achieves a near-zero-cost opportunity for high-quality machine learning theory learning.

How to Join This Study Group

Interested readers can comment on or send a direct message to the organizer on the original Reddit post, indicating their approximate time zone and available commitment. Once enough people have gathered, the organizer will formally establish the group and propose a schedule. The Discord invitation link has also been shared in the post.

For those looking to strengthen their machine learning theory foundations, this is a rare opportunity — quality textbook, study partners, and a structured pace of progress all in one. In an era where everyone is chasing the latest large model trends, taking the time to sit down with a group and work through proofs and derivations may well be the essential path to genuine professional expertise. The current industry demand for AI talent is shifting from "can call an API" to "understands the underlying principles" — talent that can analyze and design algorithms from first principles holds irreplaceable competitive advantages in core roles such as large model training optimization, novel architecture design, and reliability evaluation.