RNN
A Recurrent Neural Network (RNN) is a class of deep learning models designed for processing sequential data. Its defining characteristic is recurrent connections between neurons, which propagate hidden states across time steps to capture contextual dependencies within sequences. RNNs are widely applied in natural language processing, speech recognition, and time-series forecasting. Basic RNNs suffer from vanishing and exploding gradient problems, which led to the development of improved variants such as LSTM (Long Short-Term Memory) and GRU for more effective modeling of long-range dependencies.
Core Facts
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
The key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
Through the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
Timeline (last 90 days)
Classic experiments show that a plain RNN is almost unable to correctly capture dependencies when the gap exceeds about 10-20 words.
Models in the RNN era read word by word strictly from left to right without looking back, causing severe decay of contextual information as the hidden state advances, i.e., the long-term dependency problem.
The self-attention mechanism captures dependencies of arbitrary distance in a single forward pass by dynamically computing association weights between an element and all other elements in the sequence, overcoming the long-range dependency difficulties caused by vanishing gradients in RNNs/LSTMs.
The gradient explosion problem is particularly prominent in RNN/LSTM-based translation models, because backpropagation over long sequences tends to accumulate excessively large gradients.
RNNs store memory in a recurrent hidden state that is passed forward step by step through time.
A model can have on the order of O(N²) parameters yet carry only on the order of O(N) cross-time state, creating a mismatch between capacity and memory.
The study claims to have improved the training speed of nonlinear RNNs on time series of chaotic dynamical systems by more than two orders of magnitude (>100×).
The forward pass of an RNN is inherently sequential, with each step's computation depending on the output of the previous step, and computational complexity growing linearly with sequence length T as O[T].
RNNs prior to the Transformer could only process words sequentially one at a time, whereas the Transformer can process entire sequences in parallel, greatly improving training efficiency.
RNNs must compute sequentially step by step, making parallelization difficult and limiting training and inference efficiency.
27 more timeline events
All Facts (20)
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
90%VerifiedThe key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
90%VerifiedThrough the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
85%VerifiedThe self-attention mechanism allows the model to simultaneously reference the contextual information of all other tokens in a sequence when processing each token, rather than processing sequentially as RNNs do
80%VerifiedThe self-attention mechanism of the Transformer allows the model to directly compute dependencies between any two positions in a sequence, breaking through the parallelization bottleneck caused by the sequential computation of RNNs/LSTMs.
80%VerifiedThe self-attention mechanism is capable of capturing dependencies between any two positions in a sequence.
80%VerifiedRNNs suffer from vanishing/exploding gradient problems, where gradients propagating across a large number of time steps decay or grow exponentially, making it difficult to learn long-range dependencies.
75%VerifiedDavid Ha and Schmidhuber published the paper "World Models" in 2018, demonstrating how an agent can complete reinforcement learning training within dreams
75%VerifiedEach Transformer layer contains two core sub-modules: multi-head self-attention and a feed-forward neural network.
75%VerifiedTransformer achieves efficient modeling of long texts through its self-attention mechanism.
70%VerifiedThe Transformer architecture was proposed in 2017 and has largely supplanted RNNs in the NLP domain.
70%VerifiedBefore the advent of Transformer, the field of natural language processing relied primarily on RNNs and LSTMs to process sequential data, architectures that had to process text word by word in sequence and could not perform parallel computation
70%VerifiedCompared to traditional recurrent neural networks (RNNs), which tend to forget earlier content when processing long texts, Transformers can analyze multiple parts of a text simultaneously.
65%VerifiedTransformer的核心创新是用自注意力机制替代RNN的时序处理方式，实现了高度并行化训练，但放弃了RNN的隐状态跨时间步传递机制
65%VerifiedLSTM solves the vanishing gradient problem that standard RNNs encounter when processing long sequences by introducing three control mechanisms: the forget gate, the input gate, and the output gate
65%UnverifiedGoogle的Quick, Draw!项目于2016年发布，是一个基于循环神经网络（RNN）的涂鸦识别系统
90%Partially VerifiedThe self-attention mechanism allows the model to attend to information at all positions simultaneously, eliminating the sequential dependency limitations of RNNs/LSTMs
65%UnverifiedCompared to RNNs and LSTMs, which forget earlier content in long texts, all positions in a Transformer can be computed in parallel, making training efficiency far superior to that of RNNs and LSTMs.
60%UnverifiedThe stateless nature of Transformers stems from the multi-head self-attention mechanism; model weights are strictly frozen during inference, and there is no hidden state transfer across requests.
60%UnverifiedRNN must process inputs sequentially by time step and cannot fully utilize the parallel computing capabilities of modern GPUs, which is the fundamental reason Transformer replaced it.
60%