[KongchangAI]
ModelSALU / SALU模块

SaluNet

A normalization-free neural network architecture based on learnable element-wise geometric transformations (SALU modules) that replaces BatchNorm/LayerNorm with two learnable scalars per layer, does not rely on batch statistics, improves training stability under small batch sizes, and eliminates cross-device synchronization overhead.

Timeline (last 90 days)

Nov 9

The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces normalization layers with a learnable element-wise transformation, introduces only 2 learnable scalars per layer, and does not compute the mean and variance of activations.

Unverified50%
Nov 9

SaluNet's training was completed on a single shared NVIDIA A40 (24 GB).

Unverified50%
Nov 9

The paper's authors explicitly state that they hope to add matched baseline comparisons and more random seeds, large-batch ImageNet training of SALU+SWALU, and repeated experiments on small-batch ResNet-18.

Unverified50%

All Facts (3)

Source Articles