FlexAttention

A programmable attention interface introduced in PyTorch 2.5 that allows users to customize attention score modification and masking logic through score_mod and mask_mod hooks, while automatically generating fused, efficient kernels at the underlying level, combining flexibility with hardware acceleration.