Modular Arithmetic
Modular arithmetic encompasses mathematical tasks based on integer modular operations, such as computing (a+b) mod p. Its clear structure, non-trivial algebraic properties, and ease of variable control make it a standard benchmark task for studying the grokking phenomenon in neural networks.
Timeline (last 90 days)
The grokking phenomenon was initially widely observed on modular arithmetic tasks.
The study systematically scanned 384 hyperparameter configurations on modular arithmetic tasks using a two-hidden-layer MLP.
The typical form of a modular arithmetic task is to give a modulus p (usually a prime such as 97 or 113) and have the model learn operations of the form (a+b) mod p, with inputs as one-hot encoded integers.
For a model to truly generalize modular arithmetic tasks, it needs to implicitly discover periodic representations such as the discrete Fourier transform, rather than memorizing the answer for every pair of inputs.
All Facts (4)
The grokking phenomenon was initially widely observed on modular arithmetic tasks.
50%UnverifiedThe study systematically scanned 384 hyperparameter configurations on modular arithmetic tasks using a two-hidden-layer MLP.
50%UnverifiedThe typical form of a modular arithmetic task is to give a modulus p (usually a prime such as 97 or 113) and have the model learn operations of the form (a+b) mod p, with inputs as one-hot encoded integers.
50%UnverifiedFor a model to truly generalize modular arithmetic tasks, it needs to implicitly discover periodic representations such as the discrete Fourier transform, rather than memorizing the answer for every pair of inputs.
50%