[KongchangAI]
ConceptScalable Adaptive Linear Unit

SALU

SALU (Scalable Adaptive Linear Unit) is a learnable element-wise transformation module used in place of BatchNorm/LayerNorm; it introduces only two learnable scalars per layer, does not rely on batch statistics, and handles feature scale through geometric deformation rather than statistical standardization.

Timeline (last 90 days)

Nov 9

The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces normalization layers with a learnable element-wise transformation, introduces only 2 learnable scalars per layer, and does not compute the mean and variance of activations.

Unverified50%
Nov 9

On CIFAR-100 (ResNet-18), SALU achieves 83.42% Top-1 at BS=512 and still retains 76.23% Top-1 at BS=1, whereas standard BatchNorm collapses at BS=1.

Unverified50%
Nov 9

SALU achieves 70.80% Top-1 on CIFAR-100 (ViT).

Unverified50%
Nov 9

When trained from scratch on ImageNet-1K, SALU reaches 80.16% Top-1 with ResNet-50 and 80.28% Top-1 with ViT-B/16, using only BS=128.

Unverified50%
Nov 9

SALU's element-wise transformation depends only on each element's own learnable parameters, and neither the forward nor the backward pass requires any aggregation of statistics across samples or devices.

Unverified50%
Nov 9

The 70.80% for ViT on CIFAR-100 and the results on ImageNet come from different settings, so cross-task comparisons should be made with caution.

Unverified50%

All Facts (6)

Source Articles