Universal Transformer
An enhanced Transformer architecture that introduces recurrent mechanisms along the depth dimension, allowing the model to apply the same layer repeatedly. This design enables adaptive computation and stronger reasoning capabilities compared to standard Transformers.
Timeline (last 90 days)
ALBERT让所有层共享同一组参数，Universal Transformer在深度方向复用同一层，这些工作表明Transformer不同层学到的表示存在大量冗余。
The architectural inspiration for Looped Language Models comes from earlier works such as the Universal Transformer and DEQ (Deep Equilibrium Model).
Universal Transformer had already explored cross-layer weight sharing (parameter sharing).
Universal Transformer uses weight tying to allow all recurrent steps to reuse the same set of parameters, increasing computational depth without increasing the parameter count.
The Universal Transformer proposed by Dehghani et al. in 2019 was the first to design Transformer layers as repeatable recurrent units, and introduced an adaptive computation time mechanism to dynamically determine the number of recurrent iterations.
All Facts (5)
ALBERT让所有层共享同一组参数，Universal Transformer在深度方向复用同一层，这些工作表明Transformer不同层学到的表示存在大量冗余。
50%UnverifiedThe architectural inspiration for Looped Language Models comes from earlier works such as the Universal Transformer and DEQ (Deep Equilibrium Model).
50%UnverifiedUniversal Transformer had already explored cross-layer weight sharing (parameter sharing).
50%UnverifiedUniversal Transformer uses weight tying to allow all recurrent steps to reuse the same set of parameters, increasing computational depth without increasing the parameter count.
50%UnverifiedThe Universal Transformer proposed by Dehghani et al. in 2019 was the first to design Transformer layers as repeatable recurrent units, and introduced an adaptive computation time mechanism to dynamically determine the number of recurrent iterations.
50%