Deriving the Transformer from Scratch: You Could Have Invented It from Memory

Why Say "You Could Have Invented the Transformer from Memory"?

Recently there was a highly discussed article on Hacker News with a very direct title: "You could have invented transformers from memory". The core point it wants to convey is not complicated — the Transformer is not some magical architecture that fell from the sky, but an engineering choice that naturally grew out of existing concepts, step by step. If you understand what problem the attention mechanism is meant to solve, and then break it down into matrix multiplication and weighted sums, you'll find that the entire structure can actually be "reinvented".

This perspective is very valuable for learners. Many tutorials start directly with nn.MultiheadAttention, making people mistakenly believe they must memorize some API to understand the model. But the true path to understanding is exactly the opposite: first there is a need, then there is a design.

Starting from the Bottleneck of RNNs

To understand why the Transformer looks the way it does, we must first return to what it replaced. Recurrent Neural Networks (RNNs) process sequences step by step in time, where the hidden state at step t depends on step t-1. This brings two hard limitations: first, it cannot be parallelized, and training speed is bottlenecked by sequence length; second, long-range dependencies decay, and information from early steps is mostly gone by the time it reaches later ones.

LSTM and GRU try to alleviate forgetting with gating mechanisms, but they are still fundamentally sequential computations. The real breakthrough is not making the RNN better, but asking a different question: can every position in the sequence directly see all other positions?

Once this question is posed, the attention mechanism becomes the natural answer.

LSTM (Long Short-Term Memory) uses three gating mechanisms — the forget gate, input gate, and output gate — to let the network learn to selectively retain or discard historical information, thereby alleviating the long-range forgetting problem caused by vanishing gradients. GRU (Gated Recurrent Unit) is a simplified version of LSTM, merging the three gates into two: the update gate and the reset gate. It has fewer parameters and trains faster, with performance close to LSTM on many tasks. However, both retain a structure that computes step by step in time; step t must wait for step t-1 to finish, so parallelization remains unsolved. This fundamental limitation shows that as long as the recurrent structure is retained, no matter how elaborate the gating design, it cannot simultaneously solve both parallelism and long-range dependencies.

Attention Mechanism: Explainable in One Sentence

The intuition of attention is: when each word understands itself, it should refer to information from other words in the sentence, and how much it refers to is determined by relevance. Mathematically, this involves three things — Query, Key, and Value.

• Query: "what am I looking for" at the current position
• Key: "what label do I have" at each position
• Value: "what information do I actually provide" at each position

Take the dot product of the Query with all Keys to get relevance scores, normalize them into weights with softmax, and then perform a weighted sum over the Values. Written as a formula, this is softmax(QKᵀ/√d)V. There is nothing mysterious about this expression; it is just a weighted average.

So-called "inventing from memory" means: as long as you start from the requirement of "letting every position directly access global information" and derive it to a weighted sum, you have already reproduced the core of attention.

Multi-Head, Positional Encoding, and Residuals: Engineering Inevitabilities

Single-head attention can only capture one pattern of relevance. Multi-head attention simply runs multiple sets of Q/K/V in parallel, allowing the model to simultaneously focus on different dimensions such as syntax, semantics, and coreference. This is not a theoretical necessity, but a natural extension in expressive capacity.

Positional encoding solves another problem: attention itself is insensitive to order, and shuffling word order does not change the result. Position information must be explicitly injected, whether through sinusoidal encodings or learnable embeddings.

Residual connections and LayerNorm are not decoration either. Deep networks are generally hard to train; residuals give gradients a shortcut, and LayerNorm stabilizes distributions. These components were already standard in the ResNet era, and the Transformer simply assembled them in.

LayerNorm (Layer Normalization) differs from BatchNorm in that it normalizes all feature dimensions within a single sample, rather than across the batch dimension. This makes it more stable in scenarios with variable sequence lengths and small batches (such as NLP), and it also avoids the problem of inconsistent statistics between training and inference. Residual connections were proposed by ResNet in 2015. The core idea is to let network layers learn a "residual" rather than a complete mapping, so that gradients can flow directly back through the identity path, enabling the training of deep networks with hundreds of layers. The Transformer combines these two components into the standard sublayer structure of "pre-LayerNorm + residual", which is a culmination of deep learning engineering practice rather than a completely new invention.

Breaking the Transformer into Reproducible Building Blocks

If you were to rebuild a Transformer from memory, the path would roughly be:

1. Input embedding + positional encoding
2. Multi-head self-attention
3. Feed-forward network (two-layer MLP)
4. Residual + LayerNorm wrapping each sublayer
5. Stack N layers
6. Output head

The encoder is responsible for understanding the input, and the decoder is responsible for generating the output, connected in the middle by cross-attention. None of this structure is "must be this way"; each part can be traced back to a specific constraint. Understanding the constraints is more important than memorizing the structure.

What the original author wants to emphasize is precisely this derivability: the Transformer is not a spell to be recited, but a reasoning process you can walk through again. For people who want to truly master it, this is far more useful than memorizing formulas from the paper.

One-line Summary

The Transformer is not magic, but a reproducible engineering choice derived from the bottlenecks of RNNs; understanding the constraints matters more than memorizing the structure.

Paragraph Summary

The article points out that the Transformer is not a magical architecture that fell from the sky, but an engineering choice derived step by step from the sequential bottleneck of RNNs. RNNs process sequences step by step in time, cannot be parallelized, and suffer from decaying long-range dependencies. The gating mechanisms of LSTM and GRU only alleviate this rather than solve it. The real breakthrough is asking a different question: let every position directly access global information, making the attention mechanism the natural answer. Its core formula, softmax(QKᵀ/√d)V, is essentially a weighted sum. Multi-head attention, positional encoding, residual connections, and LayerNorm are not theoretical necessities, but natural extensions addressing concrete constraints such as expressive capacity, missing order information, and deep-network training. The author argues that re-deriving the structure from requirements allows people to truly master the Transformer better than memorizing APIs and paper formulas.