GPU training optimization
FlashAttention 1–4: How IO-Awareness Reshaped the Attention Kernel
A technical walkthrough of FlashAttention’s four generations — from IO-aware tiling on A100, through better work …
GPU Training Optimization
2026
GPU training optimization
谁偷走了 5090 的算力和显存：一步 Transformer 训练的 roofline 侦查
在 RTX 5090 上实测一步 Transformer 训练的时间和显存，再把 RMSNorm 和 attention 逐个 op 拆开，看每一步算了多少、搬了多少、为反向存了多少。最后都落到 attention 的两个 seq × …
GPU training optimization
GPU 与 Triton 入门：从存储层级到第一个 kernel
GPU 的存储层级、SM 结构、occupancy 和 CUDA 执行层级；再用 GELU、softmax、row sum、matmul 四个例子入门 Triton，附录对照 TPU。GPU training optimization 系列 …