SALU
SALU (Scalable Adaptive Linear Unit) is a learnable element-wise transformation module used in place of BatchNorm/LayerNorm; it introduces only two learnable scalars per layer, does not rely on batch statistics, and handles feature scale through geometric deformation rather than statistical standardization.
Timeline (last 90 days)
The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces normalization layers with a learnable element-wise transformation, introduces only 2 learnable scalars per layer, and does not compute the mean and variance of activations.
On CIFAR-100 (ResNet-18), SALU achieves 83.42% Top-1 at BS=512 and still retains 76.23% Top-1 at BS=1, whereas standard BatchNorm collapses at BS=1.
SALU achieves 70.80% Top-1 on CIFAR-100 (ViT).
When trained from scratch on ImageNet-1K, SALU reaches 80.16% Top-1 with ResNet-50 and 80.28% Top-1 with ViT-B/16, using only BS=128.
SALU's element-wise transformation depends only on each element's own learnable parameters, and neither the forward nor the backward pass requires any aggregation of statistics across samples or devices.
The 70.80% for ViT on CIFAR-100 and the results on ImageNet come from different settings, so cross-task comparisons should be made with caution.
All Facts (6)
The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces normalization layers with a learnable element-wise transformation, introduces only 2 learnable scalars per layer, and does not compute the mean and variance of activations.
50%UnverifiedOn CIFAR-100 (ResNet-18), SALU achieves 83.42% Top-1 at BS=512 and still retains 76.23% Top-1 at BS=1, whereas standard BatchNorm collapses at BS=1.
50%UnverifiedSALU achieves 70.80% Top-1 on CIFAR-100 (ViT).
50%UnverifiedWhen trained from scratch on ImageNet-1K, SALU reaches 80.16% Top-1 with ResNet-50 and 80.28% Top-1 with ViT-B/16, using only BS=128.
50%UnverifiedSALU's element-wise transformation depends only on each element's own learnable parameters, and neither the forward nor the backward pass requires any aggregation of statistics across samples or devices.
50%UnverifiedThe 70.80% for ViT on CIFAR-100 and the results on ImageNet come from different settings, so cross-task comparisons should be made with caution.
50%