GELU
A smooth activation function based on the Gaussian error function. It is commonly used in modern architectures such as Transformers and performs well on NLP tasks.
Timeline (last 90 days)
Non-standard activation functions (such as Swish, variants of GELU, or custom gating mechanisms), attention mechanism variants (asymmetric Q/K/V dimensions or the introduction of additional scaling factors), and dynamic routing layers (such as the routing protocol of Capsule Networks or the gating network of Mixture of Experts) are the custom layer scenarios where initialization is most prone to problems.
GELU is used in well-known language models such as the BERT and GPT series.
In the Transformer architecture, GELU or Swish has almost become the standard configuration for the feed-forward layer.
Logistic regression uses the sigmoid function exclusively, whereas neurons in neural networks can use various activation functions such as ReLU, tanh, and GELU.
MLP层通常由两个线性变换和一个激活函数（如GELU）组成，中间隐藏维度往往是输入维度的4倍
All Facts (5)
Non-standard activation functions (such as Swish, variants of GELU, or custom gating mechanisms), attention mechanism variants (asymmetric Q/K/V dimensions or the introduction of additional scaling factors), and dynamic routing layers (such as the routing protocol of Capsule Networks or the gating network of Mixture of Experts) are the custom layer scenarios where initialization is most prone to problems.
50%UnverifiedGELU is used in well-known language models such as the BERT and GPT series.
50%UnverifiedIn the Transformer architecture, GELU or Swish has almost become the standard configuration for the feed-forward layer.
50%UnverifiedLogistic regression uses the sigmoid function exclusively, whereas neurons in neural networks can use various activation functions such as ReLU, tanh, and GELU.
50%UnverifiedMLP层通常由两个线性变换和一个激活函数（如GELU）组成，中间隐藏维度往往是输入维度的4倍
50%