Flash-Attention

An IO-aware exact attention algorithm proposed by Tri Dao et al. that reduces HBM read/write operations through tiled computation, lowering attention memory complexity from O(N²) to O(N). It has become a standard high-performance kernel for Transformer training and inference.