PBT Doesn't Accelerate Grokking—It Escapes the Hyperparameter Dead Zone

Introduction: PBT's True Role in Grokking

Recently, an experiment shared by a Reddit user (code and results open-sourced) offers a counterintuitive conclusion about the role of Population Based Training (PBT) in the grokking phenomenon: PBT does not make grokking happen faster, but it can help a model escape a hyperparameter dead zone "where no configuration will grok."

Grokking refers to a phenomenon where, after a model has long since overfit on the training set, validation accuracy suddenly jumps from random levels to near-perfect. Many prior works have attempted to use hyperparameter optimization methods like PBT to accelerate this process. But this experiment, through systematic controls, paints a more nuanced picture.

The grokking phenomenon was first systematically described by Power et al. in 2022: when training neural networks on small algorithmic datasets (such as modular addition and permutations), the model first reaches 100% accuracy on the training set, but validation accuracy remains at the level of random guessing for a long time; after training for a long while longer, validation accuracy suddenly jumps to near-perfect. This "delayed generalization" is usually accompanied by a drop in weight norm and an abrupt change in representation structure, and is regarded as an important window into understanding the implicit regularization of neural networks. Since the number of steps at which grokking occurs depends heavily on hyperparameters (especially weight decay and learning rate), researchers naturally hope to use hyperparameter optimization methods to shorten the waiting time, and PBT is one of the tools that has been tried.

Experimental Setup: Modular Addition Task and Twenty-Minute Training

The authors train a small MLP on the mod 97 modular addition task, with the training set comprising only 35%, using full-batch training for a total of 20k steps. The PBT population size is 20, and each member simultaneously searches over learning rate (lr) and weight decay (wd). The baseline is a grid search with equal compute, running 3 random seeds per configuration.

This setup is deliberately kept simple to isolate the effect of PBT. But the authors also caution: the conclusions are based on only one task, one model, and one split, and their generalizability remains to be verified.

Population Based Training (PBT) is an evolutionary hyperparameter optimization method: it maintains a group of models trained in parallel (the population), periodically eliminates poorly performing members based on validation set performance, has them copy the weights of better-performing members, and applies random perturbations to hyperparameters (such as multiplying the learning rate by 0.5 or 2). Unlike grid search or Bayesian optimization, PBT does not evaluate each configuration independently; instead, it lets hyperparameters evolve dynamically during training, so it can explore non-stationary hyperparameter trajectories within a compute budget. Its core hyperparameters include population size, exploit interval, and perturbation magnitude, and these settings significantly affect whether PBT "accelerates" or "escapes."

Key Finding 1: PBT Is Not Faster Than Grid Search

In the scenario where "the corner of the grid search already contains the answer," the best fixed configuration reaches 95% validation accuracy in only 317 ± 29 steps, whereas PBT takes an average of 2667 ± 2244 steps. On every seed, PBT is slower.

This shows that when there is a "good region" in the hyperparameter space and grid search can cover it, PBT's population exploration instead incurs additional overhead. Its advantage lies not in "acceleration" but in "escape."

Key Finding 2: Starting From the Dead Zone, PBT Can Escape While Fixed Configurations Cannot

The authors designed a "dead zone": lr between 1e-4 and 1e-3, and wd between 0.01 and 0.08. Within this range, no fixed configuration can grok (0/3 success). PBT started from the dead zone does not always succeed either: slow PBT (updating every several steps) succeeds 0/3 times, but fast PBT (updating every 100 steps, with lr multiplied by 0.5 or 2) achieves 3/3 success, averaging about 6850 steps.

The path taken by fast PBT is quite instructive: lr first drops to its lower bound, then wd rises to its upper bound, and finally lr rises again. Early validation loss favors the "low lr × high wd" combination, which happens to be the wrong direction for grokking, and only later does the signal flip.

Ablation Experiments: The Selection Mechanism Is Necessary

The authors conducted several ablation experiments:

• Random donors (copying parameters randomly rather than based on fitness selection) can still escape, but later.
• Without the selection mechanism, members grok and then lose this ability.
• Across 10 seeds, a wide random search reaches 95% accuracy with a median of 300 steps, whereas PBT starting from the dead zone requires 6650 steps.
• PBT sometimes collapses into the "lr lower bound / wd upper bound" corner and never leaves.

These results show that PBT's escape ability comes partly from its exploration mechanism, but selection pressure is crucial for maintaining the grokking state.

Hyperparameter dead zone refers to a contiguous region in hyperparameter space where no fixed configuration allows the model to successfully grok. In this experiment, the dead zone is defined by learning rate 1e-4 to 1e-3 and weight decay 0.01 to 0.08. The existence of a dead zone means that if the initial hyperparameters fall within this range, no matter how long you train or how many random seeds you try, a fixed configuration cannot generalize. The value of PBT is that it does not require the initial point to be in a good region: through periodic selection and perturbation, the population can gradually push hyperparameters away from the dead zone boundary into a region where grokking is possible. This is a completely different capability from "accelerating convergence within a good region."

Key Finding 3: The Schedule Itself Carries the Result, Not Weight Copying

An important verification is the "replay" experiment: fixing the winning lr/wd schedule, not using PBT, and running it on fresh initializations. The result: 55/60 reach 95% accuracy. In contrast, using a fixed hyperparameter equal to the geometric mean of that schedule, 0/60 succeed.

This means that it is the schedule itself that carries the result, not PBT's weight copying mechanism. PBT discovered an effective hyperparameter trajectory, but this trajectory can be reproduced independently.

Schedule replay is a causal test: the hyperparameter sequence experienced by the winning member during the PBT run (the learning rate and weight decay over time) is fully recorded, and then, without using PBT, a new model is trained from scratch using that sequence as a fixed schedule. If the replay achieves a success rate close to that of PBT, it shows that the hyperparameter trajectory itself is what matters, not PBT's weight copying or population interaction. In this experiment, the replay success rate is 55/60, whereas the fixed hyperparameter equal to the geometric mean of that schedule has a success rate of 0/60, strongly supporting the conclusion that "the schedule carries the result."

Multi-Source Perspectives and Consensus

This experiment cross-validates with community discussions about PBT. According to the original Reddit post and the author's interactive charts, the core consensus is: PBT should be regarded as a "rescue tool in a bad range," not an "accelerator."

One point of divergence concerns PBT's comparison with random search: the author notes that, in a hyperparameter range with no grokking region at all, PBT should theoretically outperform random search, but this experiment has not yet tested that scenario. Therefore, the current conclusions cannot be extrapolated to a "no-solution" range.

Practical Implications and Limitations

For practitioners, this experiment conveys several signals:

1. Do not assume by default that PBT accelerates grokking. If grid search can cover the good region, a fixed configuration may be more efficient.
2. PBT is suitable as a dead-zone escape tool. When prior knowledge suggests that the initial hyperparameters may fall in a "no-solution zone," PBT's exploration mechanism has value.
3. Schedule replay is a low-cost verification method. Once an effective schedule is found, it can be used independently of PBT.

The limitations are also obvious: a single task, a single model, a single split, only 3 to 10 seeds per test, and most p-values above 0.05. The replayed schedule is a post-hoc winner, so there is selection bias. The authors readily admit that the "completely no-grokking region" range has not been tested, and that is precisely the scenario where PBT should theoretically win.

Conclusion

This experiment reminds us that the effectiveness of hyperparameter optimization tools is highly context-dependent. PBT is not a universal accelerator, but a key that works in specific predicaments. Understanding when it is useful and when it is not is more important than blindly applying it.

The code and all results are open-sourced on GitHub, and interactive charts can be viewed on the author's website.

One-line Summary

PBT does not accelerate grokking, but it can help models escape a hyperparameter dead zone with no solution.

Paragraph Summary

A controlled experiment on the mod 97 addition task re-examines the role of PBT in grokking. The experiment finds that when grid search already covers the good region, the best fixed configuration reaches 95% validation accuracy in only 317 steps, whereas PBT takes an average of 2667 steps and is slower on every seed. However, starting from a hyperparameter dead zone where no fixed configuration can grok, fast PBT achieves 3/3 escape success, whereas fixed configurations and slow PBT both fail. Ablation experiments show that the selection mechanism is necessary for maintaining the grokking state; schedule replay experiments further prove that it is the hyperparameter trajectory itself that carries the result, not PBT's weight copying. The authors therefore propose that PBT should be regarded as a "rescue tool in a bad region" rather than an accelerator, but the conclusions are limited by a single task, model, and split.