跳到正文
AI NEWS JOURNAL
NVIDIA DeveloperAI NEWS / JOURNAL

Co-Designing Sparse Attention for Efficient Long-Context Inference

Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths,...

翻译服务暂未运行,当前显示原文。

AI-Generated Summary Sparse attention limits each query to a small subset of keys, moving the cost of scoring all keys into an indexer that still processes the full context length. The indexer operates as a batched matmul with a fused epilogue, and its arithmetic intensity scales with group size G = QH/KH and speculative draft length D+1, which must fit within one Tensor Core tile of about 200 rows. Using KH = 1 with a cross-head reduction writes one logit per key, avoiding the linear decode runtime growth seen when increasing key heads. An indexer head dimension of 128 or 256 balances Tensor Core utilization against decode memory traffic, while running the epilogue and accumulation in FP16 further improves throughput. Reducing the number of keys scored through compression, hierarchical indexing, or index sharing cuts work in both the indexer and Top-k selection without changing arithmetic intensity. Fusing Top-k into the indexer kernel eliminates logit write and read-back, and reusing the previous decode step's selection as a guess accelerates Top-k by 2.1x to 2.4x. Block-level selection with large blocks lets core attention fetch keys in bulk and share them across queries, while a large core attention group size increases block reuse across query heads. Next Steps Read the Guess-Verify-Refine Top-k paper for the decode-time selection optimization. Explore the TensorRT LLM repository to see the fused Top-k implementation. Review the selective Top-k path in FlashInfer for an open-source fused indexer and Top-k kernel. Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands of tokens to hundreds of thousands. At those lengths, attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with the context length N, and at 128K tokens it can take most of the prefill time in a layer. The model-level solution is sparse attention, a transformer technique in which each query attends to only a small subset of the context, chosen per query, instead of all of it. This cuts both the compute and the memory traffic of attention on long context. This post is a practical primer for model developers designing sparse attention that runs well on NVIDIA GPUs. It focuses on model-level sparse attention, where sparsity is trained into the model, rather than drop-in methods applied after training such as BLASST, MInference, or Quest. A production FP8 kernel is used to sweep the indexer’s design knobs (query heads, key heads, head dimension, compression ratio, and speculative-decoding depth), measure how Top-k cost scales, and compare token-level with block-level core attention. Each section explains what the hardware rewards and why, and ends with a guideline you can apply to your own design. This post follows the same co-design approach as the earlier posts on dense attention and hardware-friendly LLM design. What is sparse attention and why does it matter? Sparse attention is an AI optimization technique that limits which prior tokens each query attends to. Instead of attending to the entire context, each query attends to only the \(k\) most relevant keys, typically a few thousand. Core attention then stops scaling with context. But those \(k\) keys have to be found, and the component that finds them, the indexer, still scores all \(N\) keys. The cost does not disappear; it moves into the indexer. In Figure 1, the indexer and Top-k take 47% of the DeepSeek V3.2 prefill layer. Sparse attention has three stages (Figure 2). An indexer scores every key, Top-k keeps the \(k\) highest scores, and core attention runs softmax attention over those \(k\) keys and values. The last two stages scale with \(k\), the indexer with \(N\). The indexer design space An indexer is a small attention-like block. Query heads score the current query against a cache of indexer keys, and post-processing operations (such as a ReLU and weighted sum) turn the scores into one logit per key. Table 1 shows how recent models fill that in. Table 2 provides a key to the notation used in this post. Most designs use KH = 1, 32 to 64 query heads reduced to one score with a ReLU and weighted sum, Hsz = 128, and token or small-block selection. The measurements in the next five sections, one per design knob, show which of these are efficient on NVIDIA GPUs and which should change. How does the indexer compute? The indexer is one batched matmul with a fused epilogue (Table 3). Prefill scores all ISL query tokens at once, so the matmul is large and compute-bound. Decode scores only \(D\) + 1 query tokens per KH per step and streams the whole key cache from HBM for a few rows of logits. Counting FLOPs, \(Q\) and \(K\) reads, and logit writes gives each phase’s arithmetic intensity. In prefill the \(Q\) and \(K\) terms shrink as 1 / ISL, so at long context only the logit write survives. Three observations follow from Equations 1 and 2 shown in Figure 3: Sequence length (ISL, KVSL) and CR appear in neither equation: they scale total work but not the ratio of FLOPs to bytes. Both intensities are proportional to \(G\) = QH / KH. Reducing the \(G\) heads of a group to one logit gives \(G\) times more FLOPs per logit written. Designs without this reduction (MSA, MoBA) lose that factor. Decode intensity grows almost linearly with \(D\) + 1 and is nearly independent of Hsz, as long as all \(D\) + 1 query tokens fit in one pass over the key cache. Unless noted, measurements use an FP8 indexer kernel with 64K indexer keys (CR = 4), Hsz = 128, KH = 1, PB = 1, and DB = 128. Query heads, group size, and speculative draft length Intensity is proportional to group size, but prefill and decode respond differently. Prefill is compute-bound, so more query heads mean proportionally more time. Decode is memory-bound: reading the key cache is the main bottleneck. More query heads add FLOPs but no new bytes, so runtime stays the same. Figure 4 confirms both. Decode runtime moves only 2% while \(G\) grows 16x, because the single key cache read sets the runtime. Prefill runtime grows more slowly than \(G\) at small \(G\): doubling \(G\) from 8 to 16 costs 1.66x, not 2x. This is because the FP32 logit write happens once per logit regardless of G, and at small G it dominates. Above about \(G\) = 16, the matmul dominates and runtime grows with \(G\). Equation 2 has a second lever on decode intensity: \(D\) + 1, the query tokens scored per step. Without speculative decoding, it is 1. With speculative decoding, the model drafts \(D\) tokens and verifies them with the new token in one step. This means that the indexer scores \(D\) + 1 query tokens against the same key cache. Like extra query heads, the extra query tokens add FLOPs but no new bytes, so the draft tokens are nearly free while they fit in one pass. Figure 5 shows where this process stops. One Tensor Core tile holds up to 256 query rows. With an FP32 epilogue, register pressure limits the practical tile to about 200 rows, so one pass over the key cache can score \(G\) x (\(D\) + 1) up to about 200: six query tokens at \(G\) = 32 and 26 at \(G\) = 8. Up to that point, runtime rises only 1.37x and 1.87x while the work grows 6x and 26x. Beyond that point, the rows no longer fit in one tile, so the kernel launches more CTAs, and runtime steps up. Group size and speculation depth compete for the same tile. An FP16 epilogue widens the tile to 256 rows and gives 1.31x at \(G\) = 64, \(D\) + 1 = 4 (Figure 6). Accumulating the matmul in FP16 as well reaches 1.45x, but only if the indexer QK scaling keeps the matmul outputs within FP16 range. Guideline 1: Keep \(G\) x (\(D\) + 1) within one Tensor Core tile, about 200 rows (256 with an FP16 epilogue). Guideline 2: Run the indexer epilogue in FP16. Use FP16 for accumulation too, as long as the indexer’s QK scaling keeps the matmul outputs within FP16 range. Key heads (KH) Increasing KH at a fixed QH shrinks \(G\), lowering intensity in both prefill and decode, but the two pay differently. In decode, each additional KH means one more full read of the key cache and one more set of logits to write. The key read is most of decode’s traffic, so runtime grows linearly with KH: 7.6x from KH = 1 to 8 (Figure 7). The FLOPs stay the same, but KH times the bytes move. Prefill is less sensitive: 1.3x to 1.6x from KH = 1 to 8. The FLOPs do not change, and the extra key reads are small next to the logit write that dominates prefill traffic. What grows is the per-logit work, because each key head writes its own logit for every key. A reduced-precision epilogue (Guideline 2) and Top-k fusion (Guideline 6) both shrink this work. Guideline 3: Use KH = 1 with a cross-head reduction over the \(G\) (=QH) query heads, so one logit per key is written. Head dimension (Hsz) and the epilogue Hsz enters Equation 1 directly, so a wider head raises prefill intensity. It enters Equation 2 minimally, so decode just pays for the extra key bytes. Prefill has a second effect Equation 1 does not capture. The epilogue (ReLU and weighted sum) works on the scores, whose count does not depend on Hsz. Only the matmul FLOPs grow with Hsz. At small Hsz, prefill is bound by the epilogue on the CUDA cores and the Tensor Cores wait. Widening the head adds matmul FLOPs that fit within that waiting time, so runtime grows much more slowly than Hsz until the two balance at about Hsz = 128. Figure 8 confirms this. In prefill, Hsz = 64 to 128 costs only 1.1x to 1.2x, not 2x, and beyond 128 runtime grows with the FLOPs. In decode, runtime grows with Hsz throughout, since every extra key byte is read from HBM. Guideline 4: Use Hsz = 128 or 256. Hsz = 64 leaves the Tensor Cores waiting on the epilogue. Narrower heads cut decode bytes, so most designs settle at 128. Compression ratio (CR) With CR = 4, one indexer key stands for four tokens. CR here is the indexer’s compression ratio. Core attention can compress its own KV cache separately, which mainly saves memory capacity, while indexer CR sets runtime. CR does not appear in Equations 1 or 2, so it does not change what limits the kernel. It changes the total work, linearly in 1/CR in both prefill and decode. Figure 9 shows prefill runtime doubling with every halving of CR, since every key removed takes its share of matmul, epilogue, and logit write. Decode doubles with every halving as well, except at the smallest cache (8K keys, 1.8x), where the key read is too short to reach full HBM bandwidth. Scoring fewer keys is the only way to reduce indexer work. Table 1 shows three ways: compressed indexer keys (CSA, GLM 5.3 Flash, QSA, NSA, MoBA), hierarchical indexing (LSA-Flash), and index sharing across layers (GLM 5.2, LSA-Flash). Guideline 5: Reduce the keys the indexer scores, with compression, hierarchical indexing, and index sharing. None of these changes arithmetic intensity, so use them alongside Guidelines 1 to 4, not in place of them. Top-k selection The indexer produces one logit per indexer key. Top-k scans those logits and keeps the \(k\) largest for each query, on CUDA cores in both prefill and decode. Figure 10 shows how its cost depends on how many keys are scored and kept. Scoring more keys costs more (Figure 10, left): each doubling of the key count costs about 1.7x in prefill and 1.5x in decode, because every logit is compared. Keeping more keys costs little (Figure 10, right): 64x more \(k\) raises Top-k time under 2x in prefill and 1.7x in decode. So the lever on Top-k is the same as on the indexer, the number of keys scored, and Guideline 5 helps both. Two kernel techniques reduce Top-k cost further. First, the selected keys change little from one decode step to the next. NVIDIA Guess-Verify-Refine Top-k in TensorRT LLM uses the previous step’s result as a guess and verifies it in one pass over the logits. On DeepSeek V3.2 and DeepSeek V4 decode it is 2.1x to 2.4x faster than a standard radix select. Second, fusing Top-k into the indexer removes the logit write and read-back between them, which saves time, frees the HBM that held the logits, and saves the energy of moving them. An open source example is the selective Top-k path in FlashInfer, which scores, reduces, and filters inside one kernel and never writes the full logits matrix. Guideline 6: Treat Top-k as part of the indexer’s budget. Its cost follows the keys scored, not \(k\). Fuse it with the indexer, and in decode reuse the previous step’s result. Token-level versus block-level selection Selection granularity (refer to Table 1) indicates whether the indexer selects single tokens or blocks. This choice matters most to the core attention kernel. With token-level selection (DSA, GLM 5.2, LSA), the \(k\) selected keys and values are scattered across the KV cache. The kernel looks up an address for every selected token and fetches each one separately. Since every query token selects different tokens, two queries rarely share a fetch. With block-level selection (GLM 5.3 Flash, QSA, NSA, MSA, MoBA), the selected keys and values come in contiguous blocks. One address serves a whole block, fetched in bulk, and queries that select the same block share the fetch: the kernel works block by block, fetching each block once, computing for every query that selected it, then combining each query’s partial results. Block size then sets the cost. In Figure 11, the budget is 2,048 selected keys per query and the block size varies: 16 blocks of 128 keys, 32 of 64, or 64 of 32. FLOPs and KV bytes are the same, but every block adds a query load, a partial result, and a merge step, so the kernel’s time follows the number of blocks it processes. Halving the block size nearly doubles runtime in both prefill and decode. Head sharing matters for the same reason. In dense attention, every query token attends to every block, so a fetched block is reused across all query tokens. In sparse attention, each query token picks its own blocks, so reuse is limited to the query heads of one token that share a KV head. With a large group size, where many query heads share each KV head, a fetched block serves all of them. Guideline 7: Prefer block-level selection with large blocks. Token-level selection also works, but blocks let core attention fetch keys in bulk and share them across queries. Guideline 8: Use a large core attention group size, so that many query heads share each KV head. Get started with sparse attention co-design Sparse attention makes core attention cost-effective, but the indexer that selects the keys still touches the whole context, so that is where a sparse model’s long-context cost now lives. The eight guidelines below are the design choices that keep the indexer, Top-k, and core attention efficient on NVIDIA GPUs. Apply the eight guidelines presented in this post to your own design: Guideline 1: Keep \(G\) x (\(D\) + 1) within one Tensor Core tile, about 200 rows (256 with an FP16 epilogue). Guideline 2: Run the indexer epilogue in FP16. Use FP16 for accumulation too, as long as the indexer’s QK scaling keeps the matmul outputs within FP16 range. Guideline 3: Use KH = 1 with a cross-head reduction over the \(G\) (= QH) query heads. Guideline 4: Use Hsz = 128 or 256. Most designs settle at 128. Guideline 5: Reduce the keys the indexer scores, with compression, hierarchical indexing, and index sharing. Guideline 6: Treat Top-k as part of the indexer budget. Fuse it with the indexer, and in decode reuse the previous step’s result. Guideline 7: Prefer block-level selection with large blocks. Token-level selection also works, but blocks let core attention fetch keys in bulk and share them across queries. Guideline 8: Use a large core attention group size, so that many query heads share each KV head. About the Authors Comments

资料来源

阅读原文 ↗

相关主题

← 继续阅读全部动态