Batch Normalization
A normalization technique commonly used in deep learning that standardizes activations within a mini-batch and applies a learnable affine transformation to stabilize training and accelerate convergence.
Timeline (last 90 days)
The original paper proposed normalizing noise images using affine parameters learned from natural images, rather than letting noise images independently compute batch normalization statistics.
The reproduction results show that the dedicated batch normalization layer method, which normalizes noise images using affine parameters from natural images, outperforms the baseline approach.
Layer normalization normalizes over the feature dimension of each sample, stabilizing the training process; unlike batch normalization, it does not depend on batch size and is better suited to tasks with variable sequence lengths.
Convergence failure is a multi-cause problem; learning rate, optimizer choice, missing batch normalization, unnormalized data, and improper weight initialization can all cause the loss to fail to decrease over a long period or to oscillate violently.
Common solutions for vanishing gradients include replacing sigmoid with ReLU and its variants, batch normalization, and residual connections (the core idea behind ResNet).
All Facts (5)
The original paper proposed normalizing noise images using affine parameters learned from natural images, rather than letting noise images independently compute batch normalization statistics.
50%UnverifiedThe reproduction results show that the dedicated batch normalization layer method, which normalizes noise images using affine parameters from natural images, outperforms the baseline approach.
50%UnverifiedLayer normalization normalizes over the feature dimension of each sample, stabilizing the training process; unlike batch normalization, it does not depend on batch size and is better suited to tasks with variable sequence lengths.
50%UnverifiedConvergence failure is a multi-cause problem; learning rate, optimizer choice, missing batch normalization, unnormalized data, and improper weight initialization can all cause the loss to fail to decrease over a long period or to oscillate violently.
50%UnverifiedCommon solutions for vanishing gradients include replacing sigmoid with ReLU and its variants, batch normalization, and residual connections (the core idea behind ResNet).
50%