LSTM
Long Short-Term Memory (LSTM) is a specialized recurrent neural network architecture proposed by Hochreiter and Schmidhuber in 1997. Its core design introduces three gating mechanisms—input gate, forget gate, and output gate—along with a cell state that selectively retains or discards information. This design effectively mitigates the vanishing and exploding gradient problems present in standard RNN training, enabling the capture of long-range sequential dependencies. LSTM is widely applied in natural language processing, speech recognition, and time-series forecasting.
Core Facts
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
Through the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
The self-attention mechanism of the Transformer allows the model to directly compute dependencies between any two positions in a sequence, breaking through the parallelization bottleneck caused by the sequential computation of RNNs/LSTMs.
Timeline (last 90 days)
The technical foundations of workload prediction typically include time series analysis and machine learning models (such as LSTM, Transformer, etc.), which can predict future load changes from minutes to hours in advance.
Choosing simple models such as Ridge Regression over complex models like LSTM and XGBoost is preferable at the prototype stage, because interpretability and training stability are more important, and complex models are prone to overfitting on small-scale synthetic datasets.
On the LSTM time-series prediction task, parameters were reduced from 12,051 to 2,048, with a validation set MSE as low as 0.00006.
The self-attention mechanism captures dependencies of arbitrary distance in a single forward pass by dynamically computing association weights between an element and all other elements in the sequence, overcoming the long-range dependency difficulties caused by vanishing gradients in RNNs/LSTMs.
The combination of signal decomposition techniques (such as EMD) with deep learning (such as LSTM) has been proven to be an effective approach for improving thermal load prediction accuracy.
TCN captures long-range temporal dependencies through dilated causal convolutions without introducing future information, and exhibits higher parallel efficiency and more stable gradient propagation compared to LSTM-type models.
The gradient explosion problem is particularly prominent in RNN/LSTM-based translation models, because backpropagation over long sequences tends to accumulate excessively large gradients.
LSTMs and GRUs alleviate the vanishing gradient problem by introducing gating mechanisms (forget gate, input gate, output gate), allowing the model to determine the pace of state updates on its own.
Traditional deep learning models (such as LSTM and Transformer) have parameters that are fixed after training, and when facing regime shifts they often require retraining or online learning mechanisms to adapt.
Early seq2seq implementations were mostly based on LSTM or GRU, while modern implementations extensively adopt the attention mechanism in the Transformer architecture, which handles long-range dependencies more effectively.
29 more timeline events
All Facts (20)
The Transformer's core innovation is the self-attention mechanism, which allows models to attend to all positions in the input simultaneously rather than processing step-by-step like RNNs or LSTMs
90%VerifiedThrough the self-attention mechanism, Transformer allows the model to dynamically attend to all other tokens in the sequence when processing each token, while enabling fully parallelized training.
85%VerifiedThe self-attention mechanism of the Transformer allows the model to directly compute dependencies between any two positions in a sequence, breaking through the parallelization bottleneck caused by the sequential computation of RNNs/LSTMs.
80%VerifiedThe Transformer architecture completely abandons recurrent structures and relies solely on attention mechanisms to handle sequential relationships, resolving the inherent defects of RNNs and LSTMs including vanishing gradients and the inability to perform parallel computation.
80%VerifiedBefore the advent of Transformer, the field of natural language processing relied primarily on RNNs and LSTMs to process sequential data, architectures that had to process text word by word in sequence and could not perform parallel computation
70%VerifiedLSTM solves the vanishing gradient problem that standard RNNs encounter when processing long sequences by introducing three control mechanisms: the forget gate, the input gate, and the output gate
65%UnverifiedGNMT系统采用8层LSTM编码器和8层LSTM解码器，并引入了残差连接和注意力机制
90%UnverifiedQlib内置了多种机器学习模型（如LightGBM、LSTM、Transformer等）用于股票预测，并支持自定义因子和策略开发
85%Unverified现代病虫害预警模型常用的建模方法包括时间序列分析、随机森林、LSTM神经网络
85%UnverifiedDatabox的AI能力使用异常检测（如Isolation Forest、LSTM神经网络）和时序预测技术来主动发现数据中的模式和异常
70%Partially VerifiedThe self-attention mechanism allows the model to attend to information at all positions simultaneously, eliminating the sequential dependency limitations of RNNs/LSTMs
65%UnverifiedCompared to RNNs and LSTMs, which forget earlier content in long texts, all positions in a Transformer can be computed in parallel, making training efficiency far superior to that of RNNs and LSTMs.
60%UnverifiedGoogle's Neural Machine Translation system (GNMT) is based on a multi-layer LSTM plus attention architecture, and RNN-based models dominated the field of machine translation between 2014 and 2016
60%UnverifiedTesseract introduced an LSTM-based recognition engine starting from version 4.0, and version 5.x supports Marathi (the mar language pack).
60%UnverifiedLSTM was proposed by Hochreiter and Schmidhuber in 1997, and its gating mechanism solved the vanishing gradient problem of standard RNNs.
60%UnverifiedTraditional RNNs suffer from the vanishing gradient problem; in practice, LSTMs and GRUs are more commonly used to address this.
60%UnverifiedThe technical foundations of workload prediction typically include time series analysis and machine learning models (such as LSTM, Transformer, etc.), which can predict future load changes from minutes to hours in advance.
50%UnverifiedChoosing simple models such as Ridge Regression over complex models like LSTM and XGBoost is preferable at the prototype stage, because interpretability and training stability are more important, and complex models are prone to overfitting on small-scale synthetic datasets.
50%UnverifiedOn the LSTM time-series prediction task, parameters were reduced from 12,051 to 2,048, with a validation set MSE as low as 0.00006.
50%UnverifiedThe self-attention mechanism captures dependencies of arbitrary distance in a single forward pass by dynamically computing association weights between an element and all other elements in the sequence, overcoming the long-range dependency difficulties caused by vanishing gradients in RNNs/LSTMs.
50%