arXiv Paper: Low-Rank Bottleneck Autoencoder as Attention Replacement

A paper proposing a masked language model architecture that replaces the Transformer attention mechanism with a low-rank bottleneck autoencoder. The design incorporates a three-layer hybrid module and an iterative manifold refinement method, achieving performance approaching BERT while reducing FLOPs.