ConceptMemorization Effect

Memorization Effect

The phenomenon whereby over-parameterized neural networks can memorize random labels. However, the implicit regularization of gradient descent biases them toward first learning simple, consistent regularities and only afterward memorizing exceptional cases.

Timeline (last 90 days)

Oct 15

Although over-parameterized networks have enough capacity to memorize random labels, the implicit regularization of gradient descent favors learning simple, consistent patterns first.

Unverified50%

All Facts (1)

Unverified

Although over-parameterized networks have enough capacity to memorize random labels, the implicit regularization of gradient descent favors learning simple, consistent patterns first.

50%

Source Articles

Neural Networks Don't Need Data Cleaning: A Cognitive RecalibrationOct 15