Transformer
Transformer is a deep learning model architecture based on the self-attention mechanism, proposed by Google in 2017. It processes sequence data in parallel and captures long-range dependencies, overcoming the limitations of traditional recurrent neural networks. The architecture is widely applied in natural language processing, computer vision, speech recognition, and other domains, and serves as the foundational structure for large-scale pretrained language models such as BERT and the GPT series.
Core Facts
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
The self-attention mechanism maps inputs into three sets of vectors—Query, Key, and Value—computes dot-product similarity followed by softmax normalization, and then performs a weighted sum of the Values to generate a representation that incorporates global context.
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
Timeline (last 90 days)
A long prompt that changes every turn breaks prefix caching, because prefix caching matches top-down and stops at the first difference, so state placed at the top of the prompt loses almost all the reusable content below it.
Modern embedding models are usually based on the Transformer architecture (such as BERT and its variants), encoding text into fixed-dimensional dense vectors (common dimensions are 768, 1024 or higher).
In the Transformer architecture, GELU or Swish has almost become the standard configuration for the feed-forward layer.
VIDRAFT's VKORE software acceleration layer, without changing the chip, both increases NPU inference performance several times over and makes Transformer LoRA training run on this kind of inference-specific hardware, with numerical precision aligned with standard PyTorch at 0.9999.
VIDRAFT performed LoRA training on standard Transformer blocks across all 32 chips of the Korean NPU server, achieving 0.9999 consistency with standard PyTorch training.
reViT (Recurrent Vision Transformer) achieves the accuracy of a full-depth vision encoder at equal inference FLOPs by repeatedly applying a single Transformer block recursively, and does so without intermediate feature distillation.
Under a 96-token context, ILRN's held-out test set top-1 accuracy was 27.17%, and Transformer's was 25.83%.
The state update of recurrent networks is recursive, with serial dependencies between time steps, making it hard for GPU parallel efficiency to compete with Transformer, and this bottleneck may become more pronounced as model scale grows.
In next-word prediction comparison experiments on the CNN/DailyMail dataset, ILRN and Transformer each had about 532K parameters, were each trained on 20 million next-word targets, and were trained independently under two context windows of 96 and 256.
Under a fixed training budget, a small Transformer used all delivered bits when the latent dimension was 5, but used none at all when the latent dimension was 11.
40 more timeline events
All Facts (20)
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedThe self-attention mechanism maps inputs into three sets of vectors—Query, Key, and Value—computes dot-product similarity followed by softmax normalization, and then performs a weighted sum of the Values to generate a representation that incorporates global context.
90%VerifiedThe Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
90%VerifiedMainstream LLMs are based on the Transformer architecture proposed by Google in 2017, whose core innovation is the Self-Attention mechanism.
90%VerifiedThe Prefill phase is a compute-intensive operation, while the Decode phase is a memory-bandwidth-intensive operation.
90%VerifiedThe VRAM occupied by KV Cache grows linearly with sequence length.
90%VerifiedThe Vision Transformer (ViT) was proposed by Google in 2020, dividing images into fixed-size patches as sequence inputs to a Transformer encoder.
90%VerifiedThe self-attention mechanism of the standard Transformer architecture has a computational complexity of O(n²).
90%VerifiedThe Transformer architecture was proposed by Google in 2017.
90%VerifiedLarge language models are based on the Transformer architecture and are trained on massive amounts of text data.
90%VerifiedIn the Transformer architecture, the tokens of system instructions are placed at the very front of the attention mechanism, exerting a continuous constraining influence on all subsequently generated content.
90%VerifiedThe key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
90%VerifiedThe Transformer architecture was proposed by Google in the 2017 paper "Attention Is All You Need."
90%VerifiedThe computational complexity of the standard Self-Attention mechanism has a quadratic relationship with sequence length (O(n²)).
90%VerifiedThe Transformer architecture was proposed in 2017 by the Google Brain team in the paper "Attention Is All You Need"
90%VerifiedWhisper is an automatic speech recognition model open-sourced by OpenAI in 2022, built on a Transformer architecture and trained on 680,000 hours of multilingual audio data.
90%VerifiedVision Transformer（ViT）由Google Brain团队于2020年提出，将Transformer架构引入计算机视觉领域
90%VerifiedThe core working principle of large language models such as GPT-4, Claude, and Gemini is next-token prediction based on the Transformer architecture.
90%VerifiedThe core innovation of FlashAttention is to reduce the memory access complexity of standard attention computation from O(N²) to near-linear by controlling the data transfer pattern between the GPU's SRAM and HBM
85%VerifiedThrough the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
85%