| Goal | Recommended First Paper | |------|--------------------------| | | “Rethinking Attention with Performers” (Choromanski et al. , 2021) | | Learn the causal sparse pattern | “SCAT: Sparse Causal Attention Transformer” (Zaheer et al. , 2022) | | See a concrete hybrid | “Linear‑Sparse Transformers: Merging Performers with SCAT” (Liu et al. , 2023) |
Reading those three in order will give you the mathematical foundations, the practical sparse‑attention design, and a ready‑to‑use hybrid recipe. perverformer scat