The Phase Transition Structure of Grokking: Quantifying the Shift from Memorization to Generalization

Neural networks often undergo a delayed generalization transition after surpassing the memorization stage — a phenomenon known as "grokking." Previous research has largely focused on explaining why this transition occurs, but a systematic, quantitative account of when it happens — that is, its structure in hyperparameter space — has remained elusive. A recent arXiv paper (arXiv:2609.10657v1) addresses this gap through large-scale experiments, offering the first quantifiable answers to this question.

What Is the Grokking Phenomenon?

Grokking refers to a counterintuitive training dynamic: a model achieves near-perfect fit on the training set (i.e., memorization) early on, yet test performance remains at chance level for an extended period — until, after many additional training steps, generalization suddenly "wakes up" and rapidly improves.

This phenomenon was first widely observed on modular arithmetic tasks, which have clean, controllable structure that allows researchers to isolate confounding factors. It challenges the conventional machine learning intuition that "overfitting equals generalization failure" and has become an important lens for understanding how overparameterized networks learn.

Prior work has made theoretical progress — for instance, attributing grokking to implicit regularization that eventually drives the network toward low-complexity solutions. But theoretical explanations cannot directly answer the questions engineers care most about: given a specific hyperparameter configuration, roughly when will the generalization transition occur? Can it be predicted and controlled?

The canonical form of a modular arithmetic task is: given a modulus $p$ (typically a prime such as 97 or 113), train a model to learn operations of the form $(a + b) \mod p$. Inputs are one-hot encodings of two integers; outputs are class labels for the result. Deceptively simple, these tasks possess non-trivial algebraic structure — genuine generalization requires the model to implicitly discover periodic representations akin to the discrete Fourier transform, rather than memorizing every input pair. This is precisely why modular arithmetic has become the standard grokking benchmark: the memorization and generalization pathways are clearly distinguishable, and researchers can precisely track when a model switches from a "lookup table" strategy to a "rule extraction" strategy.

Power-Law Scaling: Data Complexity Is the Dominant Factor

The central contribution of this work is a systematic sweep of 384 hyperparameter configurations on a two-hidden-layer MLP applied to modular arithmetic tasks, from which the authors fit a power-law scaling relation for the onset time of generalization:

$$T_{\mathrm{grok}} \propto H^{-0.27}, D^{-2.04}, \eta^{-0.50}, \lambda^{-0.64}$$

Here, $H$ is model width (capacity), $D$ is data complexity, $\eta$ is learning rate, and $\lambda$ is weight decay. The fit achieves $R^2 = 0.732$, rising to $0.821$ after adding interaction terms — indicating that this scaling law carries substantial explanatory power.

The most striking finding is the hierarchy among the exponents. The exponent for data complexity reaches $-2.04$, far exceeding that for model capacity at $-0.27$. This means: doubling the data accelerates generalization roughly 4×, while doubling model width yields only about a 1.2× speedup. In other words, the dominant factor governing when the grokking transition occurs is not how large the model is, but how rich the data is.

This finding has direct practical implications. When confronting delayed generalization, blindly scaling up model parameters yields far less benefit than adding high-quality data — a result that resonates with the industry-wide consensus around "data quality first."

Power-law scaling relations of the form $y \propto x^\alpha$ appear widely in physics and complex systems research, signifying scale-invariance between variables. In the machine learning context, work from OpenAI and others on Neural Scaling Laws has shown that model loss follows power-law relationships with parameter count, data size, and compute. This paper applies that analytical framework to the temporal dimension of grokking — fitting not final performance, but the moment the generalization transition occurs. The absolute magnitude of each hyperparameter's exponent directly quantifies its contribution to accelerating or delaying the transition: the larger the absolute value, the more influential the variable. This is why the contrast between data complexity's exponent of $-2.04$ and model width's $-0.27$ is so central to the paper's message.

A Sharp Phase Boundary Near Weight Decay

Beyond the scaling law, the study also identifies a clear phase boundary: when weight decay $\lambda \gtrsim 1.0$, the system crosses from a region where grokking does not occur into one where it does. This boundary is remarkably sharp — analogous to a phase transition in physics — where small changes in configuration can produce qualitative shifts in training behavior.

The study also observes that during the transition, the weight norm follows a monotonically compressing trajectory. This aligns closely with the theoretical account of grokking as implicit regularization driving the network toward a lower-complexity solution: as training progresses, the network is gradually pushed into a simpler, more generalizable region of solution space, and the sustained shrinkage of weight norm serves as an observable signal of this process.

Viewing weight decay as a "knob" that controls the phase transition gives practitioners a concrete handle. By tuning $\lambda$, researchers can in principle actively trigger or suppress grokking, enabling more precise management of overparameterized networks' training trajectories.

Phase transitions in physics describe abrupt macroscopic changes that occur when a control parameter crosses a critical value — for example, water solidifying at 0°C. Importing this concept into machine learning implies that training dynamics can exist in qualitatively distinct "phases" — in this paper, the regions where grokking does and does not occur. Weight decay $\lambda$ acts as the control parameter: when it exceeds the critical value $\lambda \gtrsim 1.0$, the system crosses from one phase into another. The value of this phase-transition perspective lies in what it implies near the boundary: critical phenomena, where model behavior is exquisitely sensitive to parameter perturbations. This not only explains why grokking has historically been difficult to reproduce reliably, but also provides a clear operational target for systematic control.

Implications for Understanding Overparameterized Networks

This work advances a phenomenon that has long remained at the level of "qualitative observation" into the domain of "quantitative prediction." Rather than merely explaining why grokking exists, it provides a framework for estimating when the transition will occur and intervening through hyperparameter choices.

From a broader perspective, grokking research speaks to a fundamental question: how do overparameterized neural networks transition from rote memorization to genuine understanding? The scaling laws and phase structure presented here offer new quantitative foundations for answering this question, and a starting point for validating these regularities on more complex tasks and larger models.

One caveat worth noting: the current experiments are limited to two-layer MLPs and the relatively simplified setting of modular arithmetic. Whether these scaling laws generalize to large-scale Transformers or real-world tasks remains to be seen. But as a systematic quantitative characterization, this work undeniably contributes a solid piece to our understanding of generalization in deep learning.