Self-Attention Mechanism

The key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.

The Transformer architecture was proposed in 2017 by the Google Brain team in the paper "Attention Is All You Need"

The self-attention mechanism allows the model to simultaneously reference the contextual information of all other tokens in a sequence when processing each token, rather than processing sequentially as RNNs do

Each Transformer layer contains two core sub-modules: multi-head self-attention and a feed-forward neural network.

Transformers process text sequences in parallel through self-attention mechanisms and can capture long-range dependencies, unlike traditional RNNs.

The self-attention mechanism determines which other positions in the sequence each position should attend to by computing the dot-product similarity between the Query and all Keys.

Transformer的核心创新是用自注意力机制替代RNN的时序处理方式，实现了高度并行化训练，但放弃了RNN的隐状态跨时间步传递机制

Set Transformer基于自注意力机制建模行向量交互，通过去掉位置编码保持排列不变性

大模型核心原理包括Transformer架构、预训练、SFT监督微调、RLHF人类反馈强化学习等关键概念

The self-attention mechanism linearly transforms the input sequence into three matrices—Query, Key, and Value—with attention weights obtained by multiplying Q by the transpose of K, normalizing via Softmax, and then multiplying by V to produce the output.

Transformer uses Positional Encoding to retain word order information while performing parallel computation.

In the standard self-attention mechanism, each token needs to compute relevance with all other tokens in the sequence, and the computational complexity is approximately the square of the sequence length (O(n²)).

The complete self-attention pipeline is: input matrix → Q/K/V projection → Q×K^T → divide by √dk → Softmax → multiply by V → context-rich output.

The final output of self-attention is obtained by weighted summation of the Value vectors using the attention weights.

Positional encoding is used to inject word order information into the model, because the attention mechanism itself does not distinguish positional order.

TabPFN requires the entire training set to be input into the Transformer as context at once, and its computational complexity grows quadratically with the number of samples.

The self-attention mechanism of Transformers can arbitrarily establish dependencies between tokens within the entire context window, representing a fundamental gap in modeling capability compared to gzip.

When processing text, the self-attention mechanism simultaneously sees all words in the entire sequence and dynamically calculates the correlation weight between each word and every other word, using parallel computation rather than processing word by word sequentially like early RNNs.

When resolution doubles, the attention computation the model must handle often grows fourfold or more, causing elapsed time to increase non-linearly.

The self-attention mechanism of the Transformer architecture has a computational complexity of O(n²), and when n grows from 8K to 128K, the theoretical computational load increases by 256 times.