GPU training optimization
FlashAttention 1–4: How IO-Awareness Reshaped the Attention Kernel
A technical walkthrough of FlashAttention’s four generations — from IO-aware tiling on A100, through better work …
Transformer
2026
GPU training optimization
谁偷走了 5090 的算力和显存：一步 Transformer 训练的 roofline 侦查
在 RTX 5090 上实测一步 Transformer 训练的时间和显存，再把 RMSNorm 和 attention 逐个 op 拆开，看每一步算了多少、搬了多少、为反向存了多少。最后都落到 attention 的两个 seq × …