SPreCap
An extended version of PreCap that draws on the self-normalizing idea of SELU. It introduces SELU-style constants and an input gain b to adjust the mean-field gradient gain χ1, aiming for self-stabilizing signal propagation in deep networks.
Timeline (last 90 days)
SPreCap introduces SELU-style constants (zero mean, unit variance as fixed points) and adds an input gain b to set the mean-field gradient gain χ1.
The researchers found that when χ1 is close to 1, the test error decreases monotonically in all four experimental units.
When SPreCap's χ1 is adjusted to match SELU (SPreCap-cs), it is nowhere significantly worse than SELU in any experimental unit, with paired median differences ranging from −0.12 to +0.20 percentage points.
Plain ELU performs comparably to SPreCap overall, suggesting that the gain from CapExp may not come entirely from its special design but is related to the overall characteristics of the ELU family.
The original SPreCap (χ1=1.13) is significantly worse than SELU at 100 layers.
All Facts (5)
SPreCap introduces SELU-style constants (zero mean, unit variance as fixed points) and adds an input gain b to set the mean-field gradient gain χ1.
50%UnverifiedThe researchers found that when χ1 is close to 1, the test error decreases monotonically in all four experimental units.
50%UnverifiedWhen SPreCap's χ1 is adjusted to match SELU (SPreCap-cs), it is nowhere significantly worse than SELU in any experimental unit, with paired median differences ranging from −0.12 to +0.20 percentage points.
50%UnverifiedPlain ELU performs comparably to SPreCap overall, suggesting that the gain from CapExp may not come entirely from its special design but is related to the overall characteristics of the ELU family.
50%UnverifiedThe original SPreCap (χ1=1.13) is significantly worse than SELU at 100 layers.
50%