Spherical Biased ReLU Features
A random feature model commonly used in theoretical analysis: the input vector is normalized to the unit sphere, projected through a random weight matrix with an added bias, and finally passed through a ReLU activation function, corresponding to the hidden-layer structure of a two-layer neural network.
Timeline (last 90 days)
For spherical bias ReLU features of dimension d, when the initialization scale τ≥1, the spectral decay exponent α = (d+2)/(d-1), and κ_0^* = 4√2 τ(d+1)/d; under ridge leverage sampling, κ_1^* = 1.
Taking d=5 and τ=10 as an example, a trainable proportion of about 4.1% captures 90% of the reduction in the approximation bound from P=0 to P=1.
All Facts (2)
For spherical bias ReLU features of dimension d, when the initialization scale τ≥1, the spectral decay exponent α = (d+2)/(d-1), and κ_0^* = 4√2 τ(d+1)/d; under ridge leverage sampling, κ_1^* = 1.
50%UnverifiedTaking d=5 and τ=10 as an example, a trainable proportion of about 4.1% captures 90% of the reduction in the approximation bound from P=0 to P=1.
50%