Flash Attention: Fast & Memory-Efficient Transformers

Compute: each of the $N^2$ entries requires a dot product of length $d$.
Memory: both $S$ (raw scores) and $P$ (softmax weights) are $N \\times N$ and must be kept in GPU memory for the backward pass.

A presentation-ready deep dive into Flash Attention — the IO-aware algorithm that makes Transformers fast on long sequences. Covers the exact mechanism, the math, tiling & recomputation tricks, and real NLP applications.