RNN
A Recurrent Neural Network (RNN) is a class of deep learning models designed for processing sequential data. Its defining characteristic is recurrent connections between neurons, which propagate hidden states across time steps to capture contextual dependencies within sequences. RNNs are widely applied in natural language processing, speech recognition, and time-series forecasting. Basic RNNs suffer from vanishing and exploding gradient problems, which led to the development of improved variants such as LSTM (Long Short-Term Memory) and GRU for more effective modeling of long-range dependencies.
Core Facts
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
The key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
Through the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
Timeline (last 90 days)
Academia keeps proposing new optimizers and new CNN/RNN architectures, but these are rarely truly adopted by industry.
The state-transition Jacobian matrix characterizes the sensitivity of the RNN hidden state to the hidden state at the previous time step, and the norm behavior of its product determines whether gradients can be effectively propagated to distant points.
The self-attention mechanism allows the model to directly attend to relevant information at any time step in the history, without needing to pass hidden states step by step as an RNN does.
The study compared the performance of decision trees, random forests, XGBoost, RNN, DNN, and Siamese neural networks in atrial fibrillation detection
In backpropagation through time (BPTT), the longer the sequence, the more easily the repeated multiplication of gradients along the time dimension leads to explosion or vanishing, whereas independent windows cut off such long-range dependencies and instead improve robustness in cross-user scenarios.
Recurrent models must be trained on shuffled, independent-window segments; if streaming windows are fed in chronological order, the network collapses to predicting only one dominant gesture.
RNNs suffer from the long-range dependency problem: as sequences grow longer, early information tends to suffer vanishing gradients during backpropagation.
The Transformer architecture replaced the recurrent structure of RNNs, greatly improving training efficiency, and is the underlying backbone of modern large language models such as GPT and BERT.
Each Transformer block contains a self-attention layer and a position-wise feed-forward network, both surrounded by residual connections and layer normalization.
The long-range dependencies of RNNs decay, so by the time early information is passed along, little of it remains.
33 more timeline events
All Facts (20)
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
90%VerifiedThe key innovation of the Transformer is the self-attention mechanism, which allows the model to simultaneously attend to information from all other tokens in the input sequence when processing a single token.
90%VerifiedThrough the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
85%VerifiedThe core innovation of the Transformer is replacing RNN's sequential processing with a self-attention mechanism, enabling highly parallelized training, while abandoning the RNN's hidden-state cross-timestep propagation mechanism.
80%VerifiedThe self-attention mechanism is capable of capturing dependencies between any two positions in a sequence.
80%VerifiedThe self-attention mechanism allows the model to simultaneously reference the contextual information of all other tokens in a sequence when processing each token, rather than processing sequentially as RNNs do
80%VerifiedThe self-attention mechanism of the Transformer allows the model to directly compute dependencies between any two positions in a sequence, breaking through the parallelization bottleneck caused by the sequential computation of RNNs/LSTMs.
80%VerifiedRNNs suffer from vanishing/exploding gradient problems, where gradients propagating across a large number of time steps decay or grow exponentially, making it difficult to learn long-range dependencies.
75%VerifiedDavid Ha and Schmidhuber published the paper "World Models" in 2018, demonstrating how an agent can complete reinforcement learning training within dreams
75%VerifiedEach Transformer layer contains two core sub-modules: multi-head self-attention and a feed-forward neural network.
75%VerifiedTransformer achieves efficient modeling of long texts through its self-attention mechanism.
70%VerifiedRNNs are designed specifically for sequential data and pass information between time steps through hidden states.
70%VerifiedThe Transformer architecture was proposed in 2017 and has largely supplanted RNNs in the NLP domain.
70%VerifiedBefore the advent of Transformer, the field of natural language processing relied primarily on RNNs and LSTMs to process sequential data, architectures that had to process text word by word in sequence and could not perform parallel computation
70%VerifiedRNNs prior to the Transformer could only process words sequentially one at a time, whereas the Transformer can process entire sequences in parallel, greatly improving training efficiency.
65%VerifiedCompared to traditional recurrent neural networks (RNNs), which tend to forget earlier content when processing long texts, Transformers can analyze multiple parts of a text simultaneously.
65%VerifiedIn the deep learning era, end-to-end models based on CNN and RNN (such as the CRNN architecture) significantly improved OCR recognition capabilities, and in recent years the Transformer architecture has also been introduced into the OCR field.
65%VerifiedSet Transformer基于自注意力机制建模行向量交互，通过去掉位置编码保持排列不变性
65%VerifiedBecause each time step's output in an RNN depends on the hidden state of the previous step, RNNs cannot perform parallel computation and are unable to leverage the parallel processing advantages of GPUs.
65%VerifiedLSTM solves the vanishing gradient problem that standard RNNs encounter when processing long sequences by introducing three control mechanisms: the forget gate, the input gate, and the output gate
65%