Longformer: The Long-Document Transformer

Sliding-window: $O(n \\cdot w)$ — local context, the foundation for contextual representations.
Dilated sliding-window: $O(n \\cdot w)$ but receptive field $= L \\cdot d \\cdot w$ — long-range reach for free.
Global attention: $O(n \\cdot g)$, $g$ small and task-specified — whole-sequence representations for prediction.

A hands-on course for ML practitioners and NLP researchers covering Longformer (arXiv:2004.05150) end-to-end: why standard Transformers fail on long documents, how Longformer's three-part attention mechanism scales linearly, the LED variant for summarization, pretraining from RoBERTa, and empirical results on character-LM, QA, and summarization benchmarks.