SaluNet
A normalization-free neural network architecture based on learnable element-wise geometric transformations (SALU modules) that replaces BatchNorm/LayerNorm with two learnable scalars per layer, does not rely on batch statistics, improves training stability under small batch sizes, and eliminates cross-device synchronization overhead.
Timeline (last 90 days)
The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces normalization layers with a learnable element-wise transformation, introduces only 2 learnable scalars per layer, and does not compute the mean and variance of activations.
SaluNet's training was completed on a single shared NVIDIA A40 (24 GB).
The paper's authors explicitly state that they hope to add matched baseline comparisons and more random seeds, large-batch ImageNet training of SALU+SWALU, and repeated experiments on small-batch ResNet-18.
All Facts (3)
The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces normalization layers with a learnable element-wise transformation, introduces only 2 learnable scalars per layer, and does not compute the mean and variance of activations.
50%UnverifiedSaluNet's training was completed on a single shared NVIDIA A40 (24 GB).
50%UnverifiedThe paper's authors explicitly state that they hope to add matched baseline comparisons and more random seeds, large-batch ImageNet training of SALU+SWALU, and repeated experiments on small-batch ResNet-18.
50%