A Fully Visual Breakdown of Self-Attention: How Transformers Actually Understand Language

A Simple Sentence Exposes the Fundamental Flaw of Early NLP

Look at this sentence: "The animal didn't cross the street because it was too tired." Then ask one question — what does "it" refer to? The street, or the animal?

We know the answer is the animal almost without thinking. But how does that "knowing" happen? The answer is that we automatically look back at the other words and link "it" with "animal." This seemingly trivial act is precisely the core problem that the self-attention mechanism was designed to solve.

In the RNN era, models read strictly word by word from left to right, never looking back. By the time the hidden state advances to the position of "it," "animal" has already been buried in the past, and contextual information has severely decayed during transmission. This is the long-range dependency problem that plagued early natural language processing — the model can see the current word but has lost the broader context.

In a fully visualized breakdown video, Bilibili creator "Visual AI" used animation to demonstrate how Transformers completely abandon this sequential processing approach. According to the video, the core idea of self-attention is remarkably simple: instead of queuing up to process words one at a time, let every word simultaneously "query" all the other words. No word is ignored, and every word contributes meaning to the entire sentence.

The long-range dependency problem refers to the phenomenon in sequence models where, when handling long-distance dependencies, early information gradually decays or is even lost as time steps advance. In RNNs and their variants (such as LSTM and GRU), the hidden state is like a container with limited capacity — every time a new word is read in, old information must be compressed into it, and the farther the distance, the more thoroughly the original information gets "overwritten." Classic experiments show that a plain RNN can barely capture dependencies correctly once the gap exceeds about 10–20 words; even though LSTM mitigates this problem through gating mechanisms, its effective memory length is still constrained by the fixed-dimensional hidden state. This is precisely the core reason why, before self-attention emerged, tasks like machine translation and question answering performed poorly on long sentences.

From Words to Vectors: Embeddings and the Q, K, V Matrices

For the math to actually work, the first step is turning words into numbers. Each word is mapped to a dense vector, a process called embedding. You can think of it as a word's "personality fingerprint" — in this vector space, "cat" is close to "dog" but far from "satellite."

Take "the cat sat" as an example. The three words are each converted into vectors of dimension d, and stacked row by row to form a 3×4 input matrix X (3 tokens, each 4-dimensional).

Next comes the key operation of self-attention: projecting the input matrix X into three independent matrices — Query, Key, and Value. Each matrix is obtained by multiplying X with its own distinct learned weight matrix. In the video's demonstration, taking the first row of X (representing the first token) and performing a dot product with each column of the Query weight matrix fills in the first row of the Query matrix.

After repeating this operation for the remaining tokens, the exact same logic is applied to the other two weight matrices, yielding the Q, K, V trio. These three matrices are the core representations that make self-attention work.

Attention Scores: How Words "Talk" to Each Other

With Q and K in hand, we can observe their interaction. Each Query represents what a word is "looking for," and each Key represents what that word "can offer."

Multiplying the Query matrix by the transposed Key matrix generates an attention score grid. Each row represents one word asking a question of all words, and each column represents the word being evaluated. A high score means a strong association; a low or negative score means there is no meaningful connection between the words.

One phenomenon in this grid is worth noting: the scores along the diagonal tend to be high. This shows that each word naturally attends to itself, ensuring the model preserves each word's original meaning. But the off-diagonal connections are equally critical — for example, a positive score between "cat" and "sat" indicates that the model has automatically linked the subject to its action. This grid is the raw attention blueprint, but at this point it is still just a pile of unprocessed numbers.

Scaled Dot-Product: How One Division Saves Us from Vanishing Gradients

The raw scores have a hidden danger. When the hidden dimension dk is large — say 64 — the scores can soar to 20, 30, or even higher. Feeding these extreme values directly into the softmax function causes the output to become completely saturated: one token gets probability 1, all the rest get 0, gradients vanish, and the network stops learning.

The Transformer architecture's solution to this is nothing short of elegant: divide the raw scores by the square root of dk. For a 64-dimensional vector, that means dividing by 8.

The effect of this division is immediate. The scores are compressed back into a healthy range centered around 0, the probability distribution output by softmax becomes smooth, and gradients can flow smoothly during backpropagation. It is precisely this division step that gives the mechanism its name: "scaled dot-product attention."

The complete formula can be summarized as: multiply Query by the transpose of Key to get raw alignment scores, divide by the square root of the Key vector dimension to stabilize variance, and apply softmax to convert them into valid attention weights. This mathematical trick is the core engine of large language models.

The softmax function converts a set of real numbers into a probability distribution by first exponentiating each value and then dividing by the sum of all exponentiated values, ensuring the outputs are all positive and sum to 1. One of its key properties is an "amplifier" effect: when the input values differ greatly, it pushes the largest toward nearly 1 and the rest toward 0, forming a sharp distribution. This is an advantage in classification tasks, but in attention mechanisms, if the scores are too large, the softmax output becomes overly concentrated, causing the gradient of the loss function with respect to the parameters to approach 0 — the so-called "vanishing gradient" or saturation phenomenon — and the network can barely update its weights through backpropagation. Dividing by √dk is precisely what keeps the variance of the dot products at roughly 1, keeping softmax in a gradient-healthy range. This is why the operation was given its own name and became an important prerequisite for stable Transformer training.

After Softmax: The Distribution Patterns of Attention Weights

Applying softmax to the scaled scores performs two key operations: first, exponentiating the values to amplify differences, then normalizing so that all values sum to exactly 1.

Each row is a valid probability distribution, and each row sums to 1. This complete attention weight matrix visually shows how much each token attends to the other tokens. The diagonal is still the brightest — you always understand your own meaning best. But the real information lies off the diagonal: "cat" strongly attends to "sat," the subject searching for its verb. These grammatical relationships are not programmatically written rules; the model learned them purely on its own through these mathematical operations.

Weighted Sum: Every Word Gets a Brand-New Contextual Representation

Once the attention weights are ready, computing the final output follows naturally: perform a weighted sum of the Value vectors using the attention weights.

For example, the output for the first token might be 61% of its own Value, 25% of "cat"'s Value, and 14% of "sat"'s Value, all added together. The result is that every token receives an entirely new vector representation.

Conceptually, think of these output vectors this way: the output for "the" is mainly itself with a touch of "cat" mixed in; the output for "cat" is a fusion of itself and "sat"; "sat" blends heavily with itself while incorporating the surrounding context. Every word has been updated according to the context of the entire sentence.

The Essence of Self-Attention: Letting Language Understand Itself

String the whole pipeline together: input matrix → Q/K/V projection → Q×K^T → divide by √dk → softmax → multiply by V → context-rich output.

This mechanism gives every word the ability to "see the whole sentence and ask questions" — what do I need from you? When this question is scaled up to billions of parameters, language models begin to truly understand us. Self-attention doesn't just process language; it teaches language to understand itself.

At the end of the video, a thought-provoking question is raised: what if a word carries multiple meanings? For example, "bank" next to "deposit" refers to a financial institution, while next to "river" it refers to a riverbank. A single attention head can capture only one relationship at a time — one head, one meaning. To attend to all meanings simultaneously, you need multiple perspectives working in parallel. This is exactly the problem that multi-head attention solves, and it is the next stop in understanding the complete Transformer architecture.

Multi-Head Attention is an extension of the self-attention mechanism: it runs multiple independent sets of Q/K/V projections in parallel, each set called a "head," and each learns to capture different types of semantic relationships in different subspaces — for example, one head focuses on subject-verb relationships, another on modifier relationships, and a third on coreference. The outputs of all heads are concatenated and then fused through a linear transformation. The intuition behind this is: single-head attention is a pair of eyes that can only focus on one relationship at a time; multi-head is observing the same sentence from multiple perspectives simultaneously. It is not mere repetition — it gives the model greater expressive capacity, at the cost of increased parameters and computation. The original Transformer paper used 8 heads, with each head's dimension being the total model dimension divided by the number of heads, to keep the overall computational cost roughly constant.

One-line Summary

A visual breakdown of self-attention that reveals how Transformers let every word query the whole sentence to understand language.

Paragraph Summary

Starting from the ambiguity of whether "it" refers to the street or the animal, the article points out the long-range dependency flaw caused by RNN sequential processing: by the time the model reads "it," "animal" has long been buried in the past and the context has severely decayed. Self-attention solves this fundamental problem by letting every word query all words simultaneously. The article then fully breaks down the computation pipeline: first map words into embedding vectors and stack them into an input matrix X, then project out the Q, K, V trio. Multiplying Query by the transposed Key produces the attention score grid, whose diagonal is often brightest, reflecting each word's attention to itself. To prevent high scores from saturating softmax and causing vanishing gradients, the scores are divided by √dk — which is exactly where the name "scaled dot-product attention" comes from. The normalized weights are used to compute a weighted sum of the Value vectors, giving every word a new representation that fuses the context of the whole sentence. The article concludes by noting that a single head can capture only one semantic relationship, and that multiple meanings require multi-head attention.