RMSNorm
A normalization method that scales only by the root mean square, without mean subtraction or bias addition; compared with LayerNorm it is computationally simpler while remaining stable in training.
Timeline (last 90 days)
The standard Transformer (Post-LN) performs normalization after residual addition, while many modern large models such as LLaMA switch to Pre-LN or RMSNorm, moving normalization to before the residual addition.
RMSNorm replaces LayerNorm, reducing computational overhead while maintaining effectiveness.
RMSNorm 相比传统 LayerNorm 去掉了均值中心化步骤，仅保留均方根归一化，计算更轻量
Gemma 采用旋转位置编码（RoPE）、GeGLU 激活函数、RMSNorm 归一化等现代 Transformer 设计
All Facts (4)
The standard Transformer (Post-LN) performs normalization after residual addition, while many modern large models such as LLaMA switch to Pre-LN or RMSNorm, moving normalization to before the residual addition.
50%UnverifiedRMSNorm replaces LayerNorm, reducing computational overhead while maintaining effectiveness.
50%UnverifiedRMSNorm 相比传统 LayerNorm 去掉了均值中心化步骤，仅保留均方根归一化，计算更轻量
50%UnverifiedGemma 采用旋转位置编码（RoPE）、GeGLU 激活函数、RMSNorm 归一化等现代 Transformer 设计
50%