PreCap
An activation function for deep neural networks intended to maintain stable signal propagation without relying on normalization layers. Its core building block, CapExp, is equivalent to a shifted and scaled ELU. PreCap applies a soft tanh clipping before CapExp, making the output smooth, monotonic, and bounded on both sides.
Timeline (last 90 days)
PreCap is an activation function whose core building block, CapExp, is an exponential unit that switches to a linear extension once the input exceeds a threshold, equivalent to a shifted and scaled ELU.
PreCap adds a soft tanh clipping before CapExp, making the entire function's output smooth, monotonic and bounded on both sides.
In early regression experiments, PreCap appeared far superior to SiLU and GELU at a fixed learning rate, but once each activation function was given its own tuned learning rate, this advantage disappeared entirely.
PreCap is not more robust to learning rate than SELU.
The authors provide a pip-installable package containing fused CUDA kernels (implemented via torch.cuda.jiterator, requiring no CUDA toolkit) and Apple MLX kernels, tested under fp32, fp64, fp16 and bf16.
Under fp32, PreCap's CUDA kernel runs at about within 1.5× the speed of native SELU.
The experiments were tested only on MLPs, using only the MNIST and FashionMNIST datasets, with short training times and only one width tested, and have not yet involved CNN or Transformer architectures.
All Facts (7)
PreCap is not more robust to learning rate than SELU.
50%UnverifiedPreCap adds a soft tanh clipping before CapExp, making the entire function's output smooth, monotonic and bounded on both sides.
50%UnverifiedIn early regression experiments, PreCap appeared far superior to SiLU and GELU at a fixed learning rate, but once each activation function was given its own tuned learning rate, this advantage disappeared entirely.
50%UnverifiedPreCap is an activation function whose core building block, CapExp, is an exponential unit that switches to a linear extension once the input exceeds a threshold, equivalent to a shifted and scaled ELU.
50%UnverifiedThe experiments were tested only on MLPs, using only the MNIST and FashionMNIST datasets, with short training times and only one width tested, and have not yet involved CNN or Transformer architectures.
50%UnverifiedUnder fp32, PreCap's CUDA kernel runs at about within 1.5× the speed of native SELU.
50%UnverifiedThe authors provide a pip-installable package containing fused CUDA kernels (implemented via torch.cuda.jiterator, requiring no CUDA toolkit) and Apple MLX kernels, tested under fp32, fp64, fp16 and bf16.
50%