FlashAttention-3: How Hopper GPU Features Push Attention to 740 TFLOPS

Attention is the most expensive operation inside a transformer. For a sequence of length N, standard attention requires O(N squared) memory and compute, which becomes a serious bottleneck as context windows grow from 2K tokens (GPT-3, OPT) to 128K tokens (GPT-4) and beyond. FlashAttention, introduced by Tri Dao et al. in 2022 (arXiv:2205.14135), solved this by reordering the attention computation with tiling and recomputation, cutting memory usage from quadratic to linear in sequence length without approximating the result. FlashAttention-2 pushed utilization to roughly 70% of theoretical max FLOPs on A100 GPUs. But when NVIDIA released the H100 (Hopper architecture), that same algorithm only hit about 35% utilization, leaving enormous performance on the table. FlashAttention-3 (arXiv:2407.08608, presented at NeurIPS 2024) closes that gap by co-designing the algorithm around three Hopper-specific hardware features.

Why FlashAttention-2 Underperformed on H100s

The H100 introduced new matrix math units, faster memory pipelines, and native low-precision support that FlashAttention-2 could not exploit. Its kernel was designed for Ampere-era hardware, where the bottleneck was HBM bandwidth. On Hopper, the compute throughput grew faster than memory bandwidth, shifting the balance. A kernel written for A100 constraints leaves Hopper's tensor cores idle during memory-bound phases, which is why utilization dropped to 35%. FlashAttention-3 rewrites the algorithm to saturate the new hardware.

Three Hopper Primitives That Enable FA-3

FlashAttention-3 builds on three hardware features specific to the Hopper GPU architecture:

• WGMMA (Warpgroup Matrix Multiply-Accumulate): a new instruction that allows entire warpgroups (128 threads) to issue matrix multiplies cooperatively, achieving higher throughput per SM than the older WMMA instructions used on Ampere.
• TMA (Tensor Memory Accelerator): a dedicated hardware unit that handles asynchronous bulk data transfers between global memory and shared memory, freeing the SM compute pipelines from memory copy work entirely.
• FP8 low-precision: native support for 8-bit floating point arithmetic in the tensor cores, enabling roughly double the throughput of FP16 operations for workloads that can tolerate reduced precision.

Together, these three features let FlashAttention-3 overlap computation with memory movement far more aggressively than any previous version.

Pingpong Scheduling: Overlapping GEMM and Softmax

Standard attention involves two matrix multiplications (Q times K transpose, then the result times V) with a softmax in between. On previous hardware, these operations ran mostly sequentially because they compete for the same execution units. FlashAttention-3 introduces inter-warpgroup pingpong scheduling, a technique that assigns alternating tiles of work to different warpgroups within the same SM. While one warpgroup executes a GEMM (matrix multiply) on the tensor cores, another warpgroup runs the softmax on the CUDA cores. Because GEMM and softmax use different hardware units, they can overlap nearly completely. This producer-consumer pipeline keeps both the tensor cores and CUDA cores busy simultaneously, eliminating the idle bubbles that dragged down FlashAttention-2 on Hopper.

Performance Results

The gains from these optimizations are substantial. In FP16 mode, FlashAttention-3 is 1.5 to 2.0 times faster than FlashAttention-2 on H100 GPUs, reaching up to 740 TFLOPS. That corresponds to roughly 75% of the H100's theoretical maximum throughput, more than doubling the utilization that FlashAttention-2 achieved on the same hardware.

With FP8 precision enabled, FlashAttention-3 reaches close to 1.2 PFLOPS. The FP8 implementation also includes techniques to reduce numerical error: it achieves 2.6 times smaller error than a baseline FP8 attention implementation. This matters because naive FP8 computation can accumulate rounding errors in the softmax normalization, potentially degrading model quality. FA-3 addresses this through careful rescaling within the tiled computation.

The Bigger Picture: Memory Pressure and Long Contexts

FlashAttention's importance extends beyond raw speed. During inference, the KV cache grows with output length while model weights stay fixed, making attention the primary source of memory pressure at long context lengths. By reducing attention's memory footprint from quadratic to linear, FlashAttention made 128K-token context windows practical. FlashAttention-3 continues this trajectory by making those long contexts faster to process on the latest hardware.

The research trajectory has continued beyond FA-3. FlashAttention-4 (arXiv:2603.05451, 2026) further co-designs the algorithm and kernel pipelining to account for asymmetric hardware scaling, where compute throughput grows faster than memory bandwidth with each GPU generation. The FlashAttention family, available at the Dao-AILab GitHub repository (github.com/Dao-AILab/flash-attention), has become foundational infrastructure for transformer training and inference at scale.

For practitioners working with large language models, the takeaway is straightforward: if you are running on H100 GPUs, FlashAttention-3 should be your default attention kernel. It delivers the throughput that Hopper hardware was designed for, and its FP8 path offers a practical route to even higher performance with controlled numerical error. On PromptingIndex, we track these infrastructure-level advances because they directly shape what context lengths and model sizes are economically viable for the prompting techniques we cover.