PreCap Activation Function: Normalization-Free 100-Layer MLP Matches SELU

Introduction: An Alternative Approach to Normalization in Deep Networks

In deep fully connected networks, normalization layers (like BatchNorm) and skip connections have become almost standard, but they also introduce additional complexity and computational overhead. Is it possible to design an activation function that inherently maintains stable signal propagation through the network, thereby eliminating the dependence on normalization? The PreCap activation function, released by an independent researcher on Reddit, is a systematic attempt in this direction. The study employed a rigorous preregistration process, publicly released all code and data, and even honestly reported negative results.

What is PreCap? From CapExp to Self-Normalization

CapExp: A Bounded Exponential Unit

The core building block of PreCap is CapExp. It is essentially an exponential unit, but when the input exceeds a certain threshold, it switches to a tangent (i.e., linear) extension. The researcher points out that CapExp is exactly equivalent to a shifted and scaled ELU. This design keeps the function bounded in the positive region, avoiding exponential explosion.

PreCap adds a soft tanh clip before CapExp, making the entire function smooth, monotonic, and bounded on both sides. SPreCap further introduces SELU-style constants (setting zero mean and unit variance as fixed points) and adds an input gain b to set the mean-field gradient gain χ1.

ELU and SELU: ELU (Exponential Linear Unit) is a classic member of the activation function family, defined as output equal to input when the input is positive (linear), and α(e^x - 1) when the input is negative. Its characteristics are a smooth negative region with a lower bound of -α, which helps mitigate vanishing gradients. SELU (Scaled ELU) multiplies ELU by a scaling factor λ (approximately 1.0507) and fixes α≈1.6733, so that when the input follows zero mean and unit variance, the output also approximately maintains zero mean and unit variance, thus possessing self-normalizing properties. The "S" in SPreCap borrows this self-normalization idea from SELU. Understanding the construction of ELU/SELU helps grasp why CapExp is described as a "shifted and scaled ELU," and how SPreCap pursues similar self-stabilizing propagation through constant settings.

Key Parameter χ1

χ1 is key to understanding SPreCap's behavior. It measures the degree of amplification or attenuation of gradients in the mean-field sense. The researcher found that when χ1 is close to 1, the test error decreases monotonically across all four experimental units. This finding directly guided subsequent fine-tuning of the χ1 value.

Mean-field theory: Mean-field analysis is a common theoretical tool for studying signal propagation in deep neural networks. It treats the pre-activations and gradients of each layer as random variables and, by assuming independence between layers, derives how their variance (or correlation) evolves layer by layer. χ1 (often denoted χ₁) is the gradient gain coefficient defined in this framework: it approximately represents the amplification factor of gradient variance between adjacent layers during backpropagation. If χ1 > 1, gradients may amplify layer by layer leading to instability; if χ1 < 1, gradients may decay. Ideally, χ1 ≈ 1 keeps the gradient signal stable along the depth direction. SPreCap adjusts χ1 through the input gain b, thereby controlling the stability of signal propagation in the mean-field sense. This concept originates from initialization theory for neural networks (such as He initialization and SELU's self-normalization analysis) and is a key perspective for understanding how deep networks can be trained without normalization.

Experimental Design and Statistical Rigor

The experimental methodology of this study deserves separate discussion. The author hash-froze the analysis scripts before each run, used different random seeds for tuning and confirmation phases, individually tuned the learning rate for each activation function, and employed paired Wilcoxon tests with Holm correction for statistical testing. This preregistration process is uncommon in the work of independent researchers and significantly enhances the credibility of the results.

Experiments were conducted on MNIST and FashionMNIST, with depths of 50 and 100 layers, width 256, using the Adam optimizer, and 15 seeds in the confirmation phase. All data, code, and preprints are publicly available.

Preregistration and hash freezing: Preregistration means that researchers publicly register their research hypotheses, experimental design, analysis methods, and primary outcome measures before data collection and analysis, to reduce bias from post-hoc selective reporting (such as p-hacking). In this study, the author hash-froze the analysis scripts before each run, i.e., computed and recorded cryptographic hash values of the scripts to ensure the analysis code was not tampered with during the experiment. This practice enhances the reproducibility and credibility of the results. Paired Wilcoxon test and Holm correction: The paired Wilcoxon signed-rank test is a non-parametric statistical method used to compare median differences between two paired samples, without relying on normality assumptions. When multiple hypothesis tests are conducted simultaneously, the family-wise error rate must be controlled; Holm correction is a step-down multiple comparison correction method that is more powerful than Bonferroni correction but equally strict. The combination of these statistical tools is relatively rigorous in empirical machine learning research, especially suitable for independent researchers to enhance the credibility of their conclusions.

Main Results: SPreCap vs. SELU

When χ1 Matches SELU: Comparable Performance

When SPreCap's χ1 is adjusted to match SELU (SPreCap-cs), it is not significantly worse than SELU in any experimental unit. The paired median differences range from −0.12 to +0.20 percentage points, a very narrow range, indicating that the two are statistically indistinguishable.

Unexpected Advantage at χ1=1.03

In a secondary test, SPreCap-e03 with χ1 set to 1.03 achieved a test error rate of 5.28% on 100-layer MNIST, compared to 6.36% for SELU. The Holm-corrected p-value was 0.0064, with 13 out of 15 seeds performing better. The author cautiously labels this as a hypothesis, as it is a secondary test requiring independent verification.

Plain ELU Also Performs Strongly

You may not have noticed that plain ELU performs comparably to SPreCap overall. This suggests that the gains from CapExp may not come entirely from its special design, but rather from the overall characteristics of the ELU family.

Negative Results: Honest Reporting of Failures

One of the most valuable aspects of this study is its complete reporting of negative results.

In early regression experiments, under a fixed learning rate, PreCap appeared far superior to SiLU and GELU. But when each activation function was given its own tuned learning rate, this advantage completely disappeared. This reminds us: comparisons across activation functions must involve fair hyperparameter tuning, otherwise conclusions may be severely biased.

Additionally, the original SPreCap (χ1=1.13) was significantly worse than SELU at 100 layers. It was precisely the gradient diagnosis of this failure case that led to the subsequent tuning of the χ1 parameter family.

PreCap is also not more robust to learning rate than SELU.

Engineering Implementation and Performance

The author provides a pip-installable package containing fused CUDA kernels (implemented via torch.cuda.jiterator, no CUDA toolkit required) and Apple MLX kernels, tested in fp32, fp64, fp16, and bf16. In fp32, the CUDA kernel runs within about 1.5x the speed of native SELU.

For a project completed by an independent researcher on consumer hardware, the engineering completeness is quite high.

Fused CUDA kernels and torch.cuda.jiterator: A fused kernel refers to combining multiple element-wise operations (such as clipping, exponentiation, linear combination in activation functions) into a single GPU kernel execution to reduce memory reads/writes and kernel launch overhead. torch.cuda.jiterator is a tool provided by PyTorch that allows users to dynamically generate CUDA kernels through Python functions without writing C++/CUDA code by hand or installing the CUDA toolkit. This greatly lowers the barrier to developing custom operators. Apple MLX: MLX is a machine learning framework introduced by Apple for Apple Silicon chips, providing a NumPy-like API optimized for the unified memory architecture. The author provides both CUDA and MLX kernels for PreCap, indicating attention to cross-hardware platform portability. fp32/fp64/fp16/bf16: These are floating-point formats of different precisions: fp32 is single precision, fp64 is double precision, and fp16 and bf16 are half precision (bf16 has a larger dynamic range). Testing multiple precisions helps evaluate the numerical stability of the activation function in mixed-precision training.

Limitations and Future Directions

The author candidly lists the limitations: only MLPs were tested, only MNIST and FashionMNIST were used, training time was short, and only one width was tested. CNN or Transformer architectures were not involved.

These limitations mean that the practical value of PreCap currently remains at the proof-of-concept stage. To determine whether it can become a general replacement for normalization, validation on more complex architectures and larger-scale datasets is needed.

Summary and Implications

PreCap research demonstrates how an independent researcher can produce credible work through preregistration, rigorous statistics, and honest reporting of negative results. Although it cannot yet replace normalization layers, the monotonic relationship between the χ1 parameter and test error, along with indications that SPreCap surpasses SELU in specific settings, provides valuable clues for activation function design.

For readers interested in fundamental deep learning research, the methodology of this project may be more worth learning from than the results themselves.

One-line Summary

An independent researcher proposes the PreCap activation function, which controls gradient gain through the χ1 parameter, matches SELU performance on deep MLPs without normalization, and honestly reports negative results through a preregistration process.

Paragraph Summary

This article introduces a novel activation function called PreCap, aimed at replacing normalization layers in deep networks. The core of PreCap is CapExp (a bounded exponential unit, equivalent to a shifted and scaled ELU), preceded by a soft tanh clip, making the output smooth, monotonic, and bounded on both sides. SPreCap further introduces SELU-style constants and an input gain b to control the mean-field gradient gain χ1. The study employs a preregistration process with hash-frozen scripts, independent seeds, paired Wilcoxon tests with Holm correction, and experiments on 50-layer and 100-layer MLPs on MNIST and FashionMNIST. Results show that when χ1 matches SELU, SPreCap performs comparably to SELU; the variant with χ1=1.03 achieves a 5.28% error rate on 100-layer MNIST, outperforming SELU's 6.36% (p=0.0064), but the author considers it only a hypothesis to be verified. The study also honestly reports negative results: after fair hyperparameter tuning, PreCap's early advantage disappears, and the original SPreCap (χ1=1.13) is significantly worse than SELU. Engineering-wise, a pip-installable package, fused CUDA kernels, and Apple MLX kernels are provided, with fp32 speed within about 1.5x of native SELU. Limitations include testing only MLPs and simple datasets, with no involvement of CNN or Transformer.