Scaled Dot-Product Attention
The standard attention computation in the Transformer, in which the dot product of queries and keys is divided by the square root of the dimension, normalized with softmax, and then used to compute a weighted sum of the values.
Timeline (last 90 days)
The scaling factor is used to prevent high-dimensional dot products from becoming too large and causing the softmax gradient to vanish.
The purpose of the scaling factor √d is to prevent the dot product results from becoming too large as the dimension increases, which would cause the softmax output to approach one-hot and the gradients to vanish.
The scaling factor (1/√d) in scaled dot-product attention only addresses the dimensionality problem, not the feature scale problem, so co-designing normalization and encoding is crucial.
All Facts (3)
The scaling factor is used to prevent high-dimensional dot products from becoming too large and causing the softmax gradient to vanish.
50%UnverifiedThe purpose of the scaling factor √d is to prevent the dot product results from becoming too large as the dimension increases, which would cause the softmax output to approach one-hot and the gradients to vanish.
50%UnverifiedThe scaling factor (1/√d) in scaled dot-product attention only addresses the dimensionality problem, not the feature scale problem, so co-designing normalization and encoding is crucial.
50%