https://noviorlu.github.io/tags/attention/Recent content in Attention on NoviorluHugo -- gohugo.iozh-CN© 2026 Noviorlu喵Mon, 05 Oct 2026 00:00:00 +0000https://noviorlu.github.io/blog/flashattention-1-to-4/Mon, 05 Oct 2026 00:00:00 +0000https://noviorlu.github.io/blog/flashattention-1-to-4/<style> .fa-intuit{margin:1.4em 0;padding:.85em 1.1em;border-radius:12px;background:rgba(var(--color-primary-500),.08);border-left:3px solid rgb(var(--color-primary-500));line-height:1.65} .fa-intuit>b:first-child{color:rgb(var(--color-primary-600))} html.dark .fa-intuit>b:first-child{color:rgb(var(--color-primary-300))} .fa-road{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:10px;margin:1.6em 0 .6em} @media (max-width:760px){.fa-road{grid-template-columns:repeat(2,minmax(0,1fr))}} .fa-step{border:1px solid rgba(128,128,128,.3);border-radius:12px;padding:.7em .85em;font-size:.82em;line-height:1.5} .fa-step b{display:block;font-size:1.1em} .fa-step i{display:block;font-style:normal;opacity:.65;margin-bottom:.5em} .fa-step span{display:block} .fa-step span+span{margin-top:.45em;color:rgb(var(--color-primary-600))} html.dark .fa-step span+span{color:rgb(var(--color-primary-300))} @media (max-width:640px){.article-content figure{overflow-x:auto}.article-content figure>svg{min-width:520px}} </style> <blockquote><p><strong>Prerequisite:</strong> <a href="https://noviorlu.github.io/blog/gpu-triton-intro/" >GPU 与 Triton 入门</a> (in Chinese) covers the GPU memory hierarchy and Triton basics; <a href="https://noviorlu.github.io/blog/gpu-training-analysis/" >谁偷走了 5090 的算力和显存</a> (in Chinese) covers FLOPs, arithmetic intensity, the roofline model and peak memory, and measures on an RTX 5090 why the $N\times N$ score matrix dominates both the runtime and the activation memory of standard attention — the two problems this post starts from.</p>