Learnable Activation Function
A deep learning technique that incorporates the internal parameters of activation functions into the gradient descent optimization process, allowing the network to automatically adjust activation function shapes during training. Common implementations include PReLU and learnable Swish, aimed at improving model expressiveness.
Timeline (last 90 days)
The core idea of trainable activation functions is to include certain parameters inside the activation function in the gradient descent optimization process, allowing the network to automatically adjust the shape of the activation function based on data.
Having activation function parameters participate in optimization may introduce new training instability factors, making the process more sensitive to learning rate and initialization.
All Facts (2)
The core idea of trainable activation functions is to include certain parameters inside the activation function in the gradient descent optimization process, allowing the network to automatically adjust the shape of the activation function based on data.
50%UnverifiedHaving activation function parameters participate in optimization may introduce new training instability factors, making the process more sensitive to learning rate and initialization.
50%