SELU
A scaled version of ELU that uses fixed scaling factors λ and α so that when the input has zero mean and unit variance the output approximately preserves zero mean and unit variance, giving it self-normalizing properties.
Timeline (last 90 days)
SPreCap introduces SELU-style constants (zero mean, unit variance as fixed points) and adds an input gain b to set the mean-field gradient gain χ1.
When SPreCap's χ1 is adjusted to match SELU (SPreCap-cs), it is nowhere significantly worse than SELU in any experimental unit, with paired median differences ranging from −0.12 to +0.20 percentage points.
In the secondary test, SPreCap-e03 with χ1 set to 1.03 achieved a 5.28% test error rate on 100-layer MNIST, versus 6.36% for SELU, with a Holm-corrected p-value of 0.0064 and 13 of 15 seeds performing better.
The authors mark the result that SPreCap-e03 outperforms SELU at χ1=1.03 as a hypothesis, because this is a secondary test that requires independent verification.
The original SPreCap (χ1=1.13) is significantly worse than SELU at 100 layers.
PreCap is not more robust to learning rate than SELU.
Under fp32, PreCap's CUDA kernel runs at about within 1.5× the speed of native SELU.
All Facts (7)
SPreCap introduces SELU-style constants (zero mean, unit variance as fixed points) and adds an input gain b to set the mean-field gradient gain χ1.
50%UnverifiedWhen SPreCap's χ1 is adjusted to match SELU (SPreCap-cs), it is nowhere significantly worse than SELU in any experimental unit, with paired median differences ranging from −0.12 to +0.20 percentage points.
50%UnverifiedIn the secondary test, SPreCap-e03 with χ1 set to 1.03 achieved a 5.28% test error rate on 100-layer MNIST, versus 6.36% for SELU, with a Holm-corrected p-value of 0.0064 and 13 of 15 seeds performing better.
50%UnverifiedThe authors mark the result that SPreCap-e03 outperforms SELU at χ1=1.03 as a hypothesis, because this is a secondary test that requires independent verification.
50%UnverifiedThe original SPreCap (χ1=1.13) is significantly worse than SELU at 100 layers.
50%UnverifiedPreCap is not more robust to learning rate than SELU.
50%UnverifiedUnder fp32, PreCap's CUDA kernel runs at about within 1.5× the speed of native SELU.
50%