FlashAttention
A technique that reduces the memory footprint and computational complexity of the Transformer attention mechanism through IO-aware tiled computation. It is one of the key technologies enabling extremely large context windows.
Core Facts
Flash Attention was proposed by Stanford's Tri Dao et al. in 2022, reducing memory complexity from O(N²) to O(N) through tiled computation and recomputation.
Flash Attention is an attention computation optimization algorithm proposed by Stanford University that processes data in chunks, keeping intermediate results in the GPU's high-speed SRAM to reduce memory bandwidth pressure and lower VRAM usage.
FlashAttention was proposed by Tri Dao et al. and improves attention computation speed by 2–4× through an IO-aware tiled computation strategy.
Timeline (last 90 days)
ModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
The maturation of inference optimization technologies such as vLLM, FlashAttention, and continuous batching has driven a sustained decline in the computational cost per token.
The RTX 5090 belongs to the Blackwell architecture, and during deployment the CUDA, PyTorch, and compiled package versions must be strictly matched; otherwise, Attention-related dependencies that accelerate inference are prone to errors.
Together AI has built an optimization system in the field of inference optimization based on high-performance operators such as FlashAttention and FlashDecoding, and has its own self-developed inference engine.
P106无法使用FlashAttention、AWQ等主流加速方案
DeepEP corresponds to NCCL, handling communication between multiple GPUs; FlashMLA corresponds to FlashAttention, used for ultra-long text processing; TileKernels corresponds to CUTLASS; DeepSelect corresponds to data pipelines such as DALI.
Tile blocking maximizes data reuse and hides memory access latency by dividing large computational tasks into small blocks that fit in shared memory or registers, and is a key technique by which high-performance libraries such as cuBLAS and FlashAttention approach peak hardware performance.
TileLang's most direct application scenarios are concentrated on core deep learning operators such as FlashAttention, GEMM (General Matrix Multiplication), and quantization and dequantization.
FlashAttention keeps intermediate results resident in SRAM through a Tiling strategy, avoiding frequent round-trips to HBM.
Concepts such as tiled matrix multiplication, memory-aware algorithms, and orthogonal updates have long existed in the relevant literature; the innovation lies in combining them and applying them to new scenarios in an engineering-ready manner.
22 more timeline events
All Facts (20)
Flash Attention was proposed by Stanford's Tri Dao et al. in 2022, reducing memory complexity from O(N²) to O(N) through tiled computation and recomputation.
90%VerifiedFlash Attention is an attention computation optimization algorithm proposed by Stanford University that processes data in chunks, keeping intermediate results in the GPU's high-speed SRAM to reduce memory bandwidth pressure and lower VRAM usage.
90%VerifiedFlashAttention was proposed by Tri Dao et al. and improves attention computation speed by 2–4× through an IO-aware tiled computation strategy.
90%VerifiedFlashAttention was proposed by Tri Dao and colleagues at Stanford University; by performing attention computation in tiles within the GPU's on-chip SRAM, it reduces memory access count by an order of magnitude and is a mathematically exact-equivalent implementation.
85%VerifiedThe core innovation of FlashAttention is to reduce the memory access complexity of standard attention computation from O(N²) to near-linear by controlling the data transfer pattern between the GPU's SRAM and HBM
85%VerifiedFlashAttention improves inference speed and maintains mathematical equivalence with standard attention computation by reordering the computation sequence and leveraging the GPU's SRAM high-speed cache to reduce memory read/write operations.
80%VerifiedFlashAttention was proposed by Tri Dao et al. in 2022.
75%VerifiedFlashAttention reduces HBM access counts from O(N²) to O(N²/M) through tiling and recomputation strategies, achieving a 2–4× speedup in long-sequence scenarios.
75%Verified自注意力机制的计算复杂度随序列长度呈近似二次方增长，序列长度为n时注意力矩阵计算量正比于n²
65%VerifiedFlashAttention (2022) reduces memory usage from O(n²) to O(n) through IO-aware tiled computation and recomputation strategies, without changing mathematical equivalence.
65%VerifiedFlashAttention通过重新设计注意力计算的内存访问模式，将显存占用从与序列长度平方成正比降低为线性关系
65%VerifiedFlashAttention fuses the multi-step computation of the attention mechanism into a single CUDA kernel, reducing memory usage from O(N²) to O(N) and achieving a 2–4x speed improvement.
65%UnverifiedFlashAttention reduces the number of reads and writes to HBM by retaining intermediate results in SRAM as much as possible, thereby breaking through the memory bandwidth bottleneck to achieve speedup.
60%UnverifiedGPUs adopt a SIMT (Single Instruction, Multiple Threads) architecture, which requires a high degree of regularity in data and tasks.
60%UnverifiedFlashAttention fuses the QKV computation, softmax, and output projection of the attention mechanism into a single kernel, reducing memory complexity from O(N²) to O(N) and achieving 2–4x end-to-end speedup.
60%UnverifiedThe breakthrough in long context windows relies on engineering techniques such as Ring Attention and FlashAttention to reduce GPU memory usage.
60%UnverifiedModernBERT replaces absolute position encoding with rotary position encoding (RoPE), uses FlashAttention for acceleration, and extends the maximum context window from BERT's 512 tokens to 8192 tokens.
50%UnverifiedThe maturation of inference optimization technologies such as vLLM, FlashAttention, and continuous batching has driven a sustained decline in the computational cost per token.
50%UnverifiedThe RTX 5090 belongs to the Blackwell architecture, and during deployment the CUDA, PyTorch, and compiled package versions must be strictly matched; otherwise, Attention-related dependencies that accelerate inference are prone to errors.
50%UnverifiedTogether AI has built an optimization system in the field of inference optimization based on high-performance operators such as FlashAttention and FlashDecoding, and has its own self-developed inference engine.
50%