SaluNet: Replacing BatchNorm and LayerNorm with Learnable Geometry

Normalization layers (BatchNorm, LayerNorm) are now virtually standard in deep learning models, but their dependence on batch size, the synchronization overhead in distributed training, and the bias in estimating statistics at inference time have long been pain points in both engineering and theory. Recently, a widely discussed paper on Reddit, "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," posed a direct question: can learnable geometric transformations completely replace feature normalization?

Core Idea: From "Moving the Distribution" to "Deforming the Function"

The authors propose the SALU (Scalable Adaptive Linear Unit) module, which replaces each normalization layer with a learnable element-wise transformation, introducing only 2 learnable scalars per layer and computing no mean or variance of activations at all. The authors summarize the difference between the three paradigms in one sentence: "Normalization moves the distribution. Derf moves the function. SALU deforms the function around a fixed center."

This means SALU does not rely on any batch statistics, so it fundamentally eliminates problems such as sensitivity to small batches and inconsistency between training and inference behavior. With only two scalars per layer, the parameter count is also extremely lightweight compared to BN's scaling and shift parameters.

The variance problem in estimating batch statistics: During training, BatchNorm standardizes activations using the mean and variance of the current mini-batch, and these two statistics are themselves random variables. When the batch size is B, the variance of the mean estimate is roughly proportional to 1/B, and the bias of the variance estimate also increases significantly as B decreases. At BS=1, the "variance" of each channel degenerates to zero (or is determined solely by a numerical stability term), so standardization effectively loses meaning and the network output fluctuates violently—this is the mechanistic root of why standard BN collapses at extremely small batch sizes. Although LayerNorm does not depend on the batch dimension, it re-centers and rescales each sample along the feature dimension, which introduces additional reduction computation and memory access overhead in large-scale Transformers, and its statistics also come from estimates over a finite number of dimensions. SALU bypasses statistic estimation entirely, so it is unaffected by the size of B and has no issue of inconsistent sources of statistics between training and inference.

Experimental Results: Stability at Small Batch Sizes Is the Highlight

The main results reported in the paper are as follows (all from the original text):

• CIFAR-100 (ResNet-18): 83.42% Top-1 at BS=512; 76.23% Top-1 is still retained at BS=1, whereas standard BN collapses under this condition.
• CIFAR-100 (ViT): 70.80% Top-1.
• ImageNet-1K (trained from scratch): ResNet-50 reaches 80.16% Top-1; ViT-B/16 reaches 80.28% Top-1, using only BS=128.
• Hardware: Training was completed on a single shared NVIDIA A40 (24 GB).

The most noteworthy result is the 76.23% retention rate on CIFAR-100 at BS=1. It is an industry consensus that BatchNorm degrades sharply at extremely small batch sizes due to the excessive variance of statistic estimates; SALU's "no statistics" design precisely bypasses this mechanism, giving it a structural advantage in small-batch settings.

Why This Matters

The problems with normalization layers go beyond accuracy. BatchNorm requires synchronizing statistics across devices in data-parallel training, which brings communication overhead; and at very small batch sizes, its performance is unreliable. LayerNorm, meanwhile, introduces extra computation in large-scale Transformer training. If SALU's approach can be more broadly validated, it may offer alternatives for the following scenarios:

• Single-GPU or small-batch training (such as the 24 GB-class GPUs commonly used by researchers);
• Deployment environments with high requirements for training/inference consistency;
• Distributed training that needs to reduce cross-device synchronization.

Handling feature scale from a "geometry" rather than "statistics" perspective also offers a new angle for research on normalization-free networks—rather than correcting statistics, let the network learn how to deform the activation function itself.

Synchronization overhead in data parallelism: In data-parallel training, the mean and variance of BatchNorm must be aggregated via all-reduce across all devices so that each GPU obtains a consistent standardization result. This means every BN layer triggers cross-device communication in both forward and backward propagation; the more layers and devices, the greater the communication volume, which can become a scaling bottleneck on large clusters. LayerNorm does not synchronize batch statistics across devices, but in tensor-parallel settings, reductions along the feature dimension may still involve inter-device communication. SALU's element-wise transformation depends only on the learnable parameters of each element itself, and neither forward nor backward propagation requires any aggregation of statistics across samples or devices, so its communication pattern is closer to an element-wise activation function than to a normalization layer.

Points That Warrant Caution

The paper is still in an early stage of dissemination, and the authors themselves are seeking collaborators, explicitly stating that they hope to add: matched baseline comparisons and more random seeds, large-batch ImageNet training with SALU + SWALU, and repeated experiments with ResNet-18 at small batch sizes. This indicates that the robustness of the current results still needs more independent reproduction to support.

In addition, the 70.80% for ViT on CIFAR-100 and the results on ImageNet come from different settings, so cross-task horizontal comparisons should be made cautiously. The specific magnitude of "BN collapse" at BS=1 and direct comparisons of SALU with other normalization-free methods (such as Fixup and GroupNorm) are not elaborated in the original post—these are key gaps for judging the method's practical value.

Fixup and GroupNorm as comparison baselines: Fixup is a class of residual network training techniques that do not rely on normalization layers; it maintains signal propagation in deep networks through specific parameter initialization and scaling strategies (such as multiplying the residual branch by a small constant and zero-initializing the last few layers), thereby training ResNet without normalization. GroupNorm divides channels into groups and normalizes within each group along the spatial dimensions and the channels in that group; it does not depend on batch size and is often used in small-batch detection and segmentation tasks. These two overlap with SALU's target scenarios: Fixup likewise pursues normalization-free training, and GroupNorm is likewise friendly to small batches. If SALU is to be compared directly with them, results should be reported under the same backbone, the same number of training epochs, and the same data augmentation, and it should be explained whether SALU's additional learnable scalars change the effective learning rate or initialization sensitivity; otherwise it is difficult to determine whether its accuracy advantage comes from the geometric deformation itself or from hyperparameter adaptation.

Conclusion

SALU, proposed by SaluNet, replaces normalization layers with a geometric deformation of two scalars per layer, delivering usable accuracy numbers on CIFAR-100 and ImageNet, with stability at small batch sizes being particularly notable. Whether it can truly become a general replacement for BatchNorm/LayerNorm depends on subsequent completion of baselines and independent reproduction. For researchers interested in normalization-free architectures, small-batch training, and training on lightweight hardware, this paper and its open-source code are worth following.

One-line Summary

SaluNet replaces normalization layers with a geometric deformation of two scalars per layer and performs stably at small batch sizes.

Paragraph Summary

The paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" proposes the SALU module, which replaces BatchNorm/LayerNorm at each normalization layer with an element-wise learnable geometric transformation, introducing only 2 scalars per layer and computing no activation statistics, thereby fundamentally avoiding small-batch sensitivity and training/inference inconsistency. Experiments show that on CIFAR-100, ResNet-18 reaches 83.42% Top-1 at BS=512 and still retains 76.23% at BS=1, whereas standard BN collapses at BS=1; on ImageNet-1K trained from scratch, ResNet-50 reaches 80.16% and ViT-B/16 reaches 80.28% (with only BS=128), all completed on a single A40. The authors are seeking collaboration on reproduction with matched baselines and more random seeds, and robustness still requires independent verification.