SwiGLU
A feed-forward network activation function that combines the Swish activation with a Gated Linear Unit (GLU). It has demonstrated performance superior to ReLU and GeLU in several large language models and has been adopted by models including LLaMA and PaLM.
Timeline (last 90 days)
In MoE architectures, the number of gating activation calls grows linearly with the number of experts, causing the proportion of SwiGLU/SiLU computation to be far higher than in dense models.
On the NVIDIA GB200 platform, full training step throughput improves by 8.0% for routed-expert SwiGLU tasks, the highest gain among the four tasks.
Different models differ in attention mechanisms, activation functions, and normalization methods (e.g., GQA, RoPE, SwiGLU), and each variant requires the runtime framework to provide corresponding operator implementations.
Agnes-3.0-Flash's feed-forward network uses SwiGLU with an intermediate dimension of 17408, with a parallel 2048-dimensional SwiGLU branch per layer
Noam Shazeer's SwiGLU activation function paper attributes the architecture's success to "divine benevolence" in the conclusion section
The model architecture is a cross-attention DiT (896 width × 16 layers) using 2D RoPE, QK-norm, SwiGLU, and adaLN-single.
The FFN feed-forward network module upscales vectors via the up layer (e.g., 4K→8K), passes them through an activation function (such as SwiGLU gated activation), and then reduces them back to the original dimension via the down layer.
The SwiGLU activation function is the feed-forward network design adopted by mainstream modern large models such as LLaMA.
SwiGLU was proposed by Google researcher Noam Shazeer in 2020, combining the Swish activation function with GLU.
SwiGLU increases the number of feed-forward layer parameters by approximately 50%, but has been proven to consistently improve model quality under the same compute budget, and is adopted by both Meta's LLaMA and Google's PaLM.
All Facts (10)
In MoE architectures, the number of gating activation calls grows linearly with the number of experts, causing the proportion of SwiGLU/SiLU computation to be far higher than in dense models.
50%UnverifiedOn the NVIDIA GB200 platform, full training step throughput improves by 8.0% for routed-expert SwiGLU tasks, the highest gain among the four tasks.
50%UnverifiedDifferent models differ in attention mechanisms, activation functions, and normalization methods (e.g., GQA, RoPE, SwiGLU), and each variant requires the runtime framework to provide corresponding operator implementations.
50%UnverifiedAgnes-3.0-Flash's feed-forward network uses SwiGLU with an intermediate dimension of 17408, with a parallel 2048-dimensional SwiGLU branch per layer
50%UnverifiedNoam Shazeer's SwiGLU activation function paper attributes the architecture's success to "divine benevolence" in the conclusion section
50%UnverifiedThe model architecture is a cross-attention DiT (896 width × 16 layers) using 2D RoPE, QK-norm, SwiGLU, and adaLN-single.
50%UnverifiedThe FFN feed-forward network module upscales vectors via the up layer (e.g., 4K→8K), passes them through an activation function (such as SwiGLU gated activation), and then reduces them back to the original dimension via the down layer.
50%UnverifiedThe SwiGLU activation function is the feed-forward network design adopted by mainstream modern large models such as LLaMA.
50%UnverifiedSwiGLU was proposed by Google researcher Noam Shazeer in 2020, combining the Swish activation function with GLU.
50%UnverifiedSwiGLU increases the number of feed-forward layer parameters by approximately 50%, but has been proven to consistently improve model quality under the same compute budget, and is adopted by both Meta's LLaMA and Google's PaLM.
50%