Vaswani
A deep learning researcher and one of the primary authors of the paper "Attention Is All You Need," in which the Transformer architecture was introduced. The work proposed replacing recurrent and convolutional networks with a model based entirely on self-attention mechanisms, which became foundational to subsequent advances in natural language processing and other domains.
Timeline (last 90 days)
Standard softmax attention (Vaswani et al., 2017) has O(n^2) time and memory complexity in the sequence length n.
Additive sinusoidal positional encoding gives different attention scores for the same offset but different absolute positions, because after expanding the dot product, cross terms related to the absolute positions remain.
Transformer最初由Vaswani等人在2017年的论文《Attention Is All You Need》中提出，用于机器翻译任务
All Facts (3)
Transformer最初由Vaswani等人在2017年的论文《Attention Is All You Need》中提出，用于机器翻译任务
95%UnverifiedStandard softmax attention (Vaswani et al., 2017) has O(n^2) time and memory complexity in the sequence length n.
50%UnverifiedAdditive sinusoidal positional encoding gives different attention scores for the same offset but different absolute positions, because after expanding the dot product, cross terms related to the absolute positions remain.
50%