Yuan Jingyang of Peking University: Sparse Attention Mechanisms Enable 10x Model Speedup | Attention

NSA (Native Sparse Attention) is a concept that researchers in natural language processing (NLP) and deep learning model optimization have repeatedly discussed and referenced, and it stands as one of the pivotal works in the Attention field.

For this conversation, we invited Dr. Jingyang Yuan from the School of Computer Science Peking University, first author of the ACL 2025 Best Paper Native Sparse Attention: Hardware-Aligned and Natively Sparse Attention, co-authored with Wenfeng Liang's team at DeepSeek.

• When next-generation models require 128k or 1M context lengths, which attention mechanism will actually survive?
• If we want models to truly "remember all history" and sustain reasoning for tens of minutes, how must Attention be restructured?
• Is there an attention architecture that can operate sparsely during pretraining while achieving 10×+ acceleration at inference without sacrificing accuracy?

Driven by curiosity about these questions, we sat down with Dr. Yuan for an in-depth discussion of the innovations and thinking behind the paper. What follows is a curated selection from the research and our conversation — reading time approximately 20 minutes.

Enjoy

Ultra-long context capability is becoming the dividing line for next-generation large model performance.

The market demands that models comprehend entire codebases, lengthy documents, and maintain coherence across tens of thousands of tokens in chain-of-thought reasoning. But as sequence lengths climb, the computational and memory access costs of standard full attention quickly become system bottlenecks. At 64k context lengths, for instance, Softmax attention often consumes 70–80% of decoding latency, capping both training and inference costs at an unsatisfying ceiling.

While numerous sparse attention approaches have emerged in recent years, most only take effect during inference, remain imperfectly aligned with modern GPU architectures, and lack end-to-end training capability — leaving theoretical gains unrealized in practice.

The core contribution of this work, NSA (Native Sparse Attention), attempts to fundamentally rewrite this structural contradiction.

Rather than retrofitting sparsity onto a full-attention model after the fact, it reorganizes information flow at the architectural level: distant tokens are aggregated into contiguous chunks, capturing global contours as "block-level summaries"; these chunks then undergo lightweight scoring to surface a handful of critical blocks, preserving important details; while local context near the current token is modeled fully through a sliding window.

These three attention paths compute in parallel, ultimately fused into a single output through a learnable gating mechanism. Meanwhile, this structure uses "contiguous blocks" as its fundamental unit of operation, with new Triton kernels (Triton being an open-source programming language and compiler) designed specifically for GPU memory characteristics and Tensor Core (NVIDIA's dedicated processing cores) scheduling, enabling genuine acceleration for sparse attention during training, prefill, and decoding.

On a 27B model combining GQA and MoE, NSA participates in computation sparsely from the pretraining stage: at 64k context, forward training reaches roughly 9× the speed of full attention, backward reaches 6×. During decoding, KV memory access drops to approximately one-tenth of baseline, corresponding to near-theoretical-limit 11.6× acceleration. Crucially, performance does not degrade due to sparsity. On general benchmarks including MMLU, GSM8K, and DROP, results are on par with full attention; on long-context tasks like LongBench and 64k Needle-in-a-Haystack, advantages are more pronounced; in AIME 24 mathematical reasoning experiments, even within 8k or 16k generation windows, the NSA-based model demonstrates more reliable chain-of-thought accuracy than full attention.

NSA outperforms multiple existing sparse attention methods

These results collectively point to a thought-provoking conclusion: for long-context models, performance and efficiency need not be zero-sum tradeoffs.

The goal should be teaching models to "spend compute where it matters."

NSA's first layer of innovation elevates sparsity from "optimization technique" to "structural principle."

Previous sparse attention approaches typically relied on heuristic rules or post-hoc pruning, with the model itself still dominated by full attention. In NSA, attention is decomposed into three parallel paths: a compression path sketching global contours, a selection path responsible for preserving critical details, and a sliding window path ensuring the model doesn't lose sensitivity to local context.

This division allows the model to understand the same context at different scales, making sparsity a natural outcome of structural learning rather than an externally imposed constraint. More importantly, the three paths coordinate through learnable gating mechanisms, enabling the model to self-shape during pretraining and gradually form optimal sparse behavior patterns.

The second layer of innovation comes from deep coupling with modern GPUs.

The model performs selection and loading only on contiguous token blocks — this block structure is natively compatible with GPU memory access patterns. The paper designs new Triton kernels tailored to GQA/MQA characteristics: multiple attention heads within the same GQA group share sparsified KV indices, and all KV blocks needed for computation are loaded into SRAM (Static Random-Access Memory) in a single, contiguous operation. This "load-by-group, compute-by-block" approach maximizes Tensor Core utilization, avoiding the hardware waste caused by "scattered memory access" in previous sparse schemes.

The third layer of innovation concerns "how to learn sparsity."

It reuses the attention distributions generated mid-process by the compression path to approximate importance for each block, and makes this process differentiable through a continuous Top-n selection mechanism, enabling direct learning of sparse patterns through backpropagation during pretraining. This means the model can automatically explore "which blocks to retain and which to ignore" across massive training corpora, without hand-designed heuristic rules and without the training instability caused by discrete selection.

The following is a curated transcript of our conversation:

Oasis Capital: Dr. Yuan, thank you so much for joining us today. Let's start from a more personal place: could you introduce your research background and what ultimately led you to "efficient attention"?

Dr. Yuan: Certainly, I'm also glad to discuss our work with everyone.

I'm currently pursuing my PhD at Peking University, in the School of Computer Science, focusing primarily on large-scale model architecture research. My own work concentrates more specifically on efficient attention. I imagine your team reached out because of my research around efficient attention, or sparse attention more broadly.

The entire research thread and original motivation actually stem from a very practical trend. As models become more intelligent, they need to process increasingly longer contexts. This means not just receiving long text inputs, but maintaining continuous reasoning capability across long sequences. And the efficiency bottleneck in long-context processing has remained a core obstacle limiting further model development.

Oasis Capital: Could we understand "efficient attention" not as an isolated technical point, but as a systemic problem that cannot be avoided in the long-context era?

Dr. Yuan: Exactly. We've repeatedly observed in our research that a major reason models struggle to scale to very long contexts comes from the inherent cost of attention. Simply put, as long as attention complexity grows quadratically with sequence length, computational cost inevitably explodes.

In dense attention, every token must compute relationships with all other tokens. Of course, we can see all these pair-wise connections, but the proportion that actually matters is quite low.

In other words, most attention computation simply isn't "worth" doing.

This was our starting point for rethinking sparse attention.

At the same time, another important observation comes from linguistics and model behavior itself. We found that within a large language model, word-to-word associations display very pronounced "sparsity" — not every word strongly connects to all others. So when a model processes a token, the context it actually depends on tends to be structural, local, or block-like, rather than an undifferentiated dense set. This sparse structure is not only a feature of language itself, but also a pattern the model gradually learns internally.

Oasis Capital: So it sounds like you're saying the statistical structure of language makes it impossible for Attention to be uniformly dense?

Dr. Yuan: You could put it that way.

In fact, from the properties of Softmax, we can reach a similar conclusion. Softmax is fundamentally a competitive activation function — after normalization, weights naturally exhibit a "few prominent, many decaying" distribution. This means attention weights spontaneously become sparse; traditional dense attention simply computes all pairs without exploiting this sparsity.

Based on these observations, we began exploring: since the effective portion of attention is naturally sparse, internal model structure is naturally sparse, and Softmax behavior is naturally sparse, why not make attention a "natively sparse" mechanism at the architectural level?

This was the motivation behind our native sparse attention mechanism.

Oasis Capital: In our discussion just now, you've already touched on the core motivation of the paper: what problem are we actually trying to solve?

But breaking the work down, what roles do NSA's three branches — compression, selection, and sliding window — each play? Why does attention need to be split into these three paths?

Dr. Yuan: Following our earlier discussion, a direct question emerges: the crux of sparse attention isn't "how to become sparse," but "how to select that small portion of tokens that should be retained."

In other words, the core of sparse attention lies not in sparsity itself, but in indexing — that is, how to find the most critical, most relevant tokens.

This step has an almost oracle-like quality, because when using dense attention for computation, the model itself already reveals which tokens should be prioritized. If you've already calculated all the dense attention scores, you've lost the very point of sparsity. So in NSA, we placed "how to build the index" at the center of our entire approach.

This is why the compression branch sits at the forefront.

Its task isn't to replace attention, but to provide a lightweight yet reliable source of scoring: in "compression attention," we perform block-level aggregation on the original sequence, obtaining coarse-grained importance score rankings for these blocks through a cheaper full attention pass. These scores won't be highly refined, but they're sufficient as "preliminary judgments" for the subsequent selection branch.

Oasis Capital: So compression isn't meant to directly produce output, but to provide a learnable, lightweight "pre-screening"?

Dr. Yuan: Right. If we relied solely on the selection branch, having the model determine each token's importance from the start would create extremely high learning difficulty. The compression output helps the model reduce search space from token-level to block-level, making the entire selection process more stable and controllable.

Before entering the true sparse branch, the second thing we need to do is use these compressed importance scores as indexing signals for sparse attention. This means we don't merely retain the most important tokens (or token blocks), but use these positions as the primary computation locations for sparse attention, concentrating limited compute there.

Oasis Capital: Why can't the compression branch simultaneously handle both "selection" and "output"? Why must there be a separate selection branch?

Dr. Yuan: This is a crucial point. The reason is that compression itself is not a reversible process — it's an information-losing operation. You can only use it to judge which blocks are important, not to recover details. Therefore, it cannot directly serve as the final attention output, because then the model would be unable to learn token-level fine-grained associations.

So we must make the selection branch an independent attention branch, providing fine-grained attention computation for truly sparse regions. Its existence ensures that while the model reduces computational load, it doesn't lose sensitivity to critical information.

Oasis Capital: What about the sliding window branch? Why is a dedicated branch for local context needed?

Dr. Yuan: This is another structural phenomenon we observed early in training: before learning long contexts, models tend to first rely on dense information within a region, similar to short-distance dependency structures in language.

If at this stage the learning network could only access sparse signals from "compression" and "selection," it would prematurely fall into a state of "over-reliance on long-distance attention, neglecting local semantics." Such a training process would be unstable.

So we introduced a third branch, "sliding window attention." Its role is very clear: in early training, provide the model with a stable, reliable local attention channel, letting it first establish the most common and robust short-range relationships in natural language. This branch essentially "catches" the model, preventing the entire system from losing direction due to signal sparsity in early training.

Oasis Capital: So this is the final NSA three-branch architecture: compression handles pre-screening, selection handles critical details, and sliding window ensures stable training.

Dr. Yuan: Yes. We also emphasize NSA's "trinomial structure" in the paper, because these three attention types serve completely different functions during training and inference — simply put:

• Compression handles "where to look";
• Selection handles "what to look at";
• Sliding Window handles "ensuring the model can successfully learn how to look."

If any path is missing, end-to-end training of native sparse attention cannot succeed — which is why previous sparse attention methods struggled with end-to-end training. They lacked a stable learning process.

So in this sense, NSA's true innovation isn't "making attention sparse," but "enabling the model to learn how to operate sparsely."

Oasis Capital: We've discussed NSA's algorithm-level design.

In the paper, you also emphasize re-examining the limitations of Flash Attention kernels. Could you elaborate on this?

Dr. Yuan: This is a frequently overlooked but absolutely central aspect of NSA. To understand it, we need to return to how Flash Attention itself was designed.

Flash Attention splits Q, K, V along the sequence dimension into blocks, then ensures high kernel throughput through block-level contiguous memory access. This structure is ideal for dense attention, because every Q head accesses the complete K-V sequence with strong data continuity and high hardware utilization.

But when you try to make attention sparse, this advantage immediately becomes a limitation.

Because the defining characteristic of sparse attention is that each q needs to access k-v blocks that aren't contiguous, but dynamically selected based on index. When Flash Attention assumes "all heads read data at the same block boundaries," it cannot accommodate sparse attention's irregular, dynamically-selected access patterns.

This problem becomes especially pronounced in NSA's "query-aware" sparse structure.

If multiple Q heads within a block each need to access different K-V regions, and these regions aren't contiguous, the system is forced to repeatedly load large amounts of non-contiguous KV blocks from memory. All sparse advantages are then offset by I/O costs amplified tens or hundreds of times.

This explains why traditional Flash Attention kernels cannot directly support NSA: their assumptions are dense, sequential, and regular; what we need is sparse, dynamic, and non-contiguous. The two are fundamentally conflicting in pattern.

Oasis Capital: So NSA's core goal is "making sparse attention actually run on hardware, and truly accelerate"?

Dr. Yuan: Yes, you could say that. The core challenge of sparse attention isn't "reducing computation," but "reducing memory access." If you reduce computation but double I/O from hunting down scattered sparse blocks, it'll never be faster than dense attention.

So NSA had to be redesigned from hardware access patterns upward. What we did is simple but critical:

We have all heads belonging to the same GQA group share the same set of KV blocks within a single decoding step. This way, you only need to load these blocks once into SRAM, and all heads can select based on them.

The intuition behind this is: unify selection at the algorithm level, unify loading at the hardware level.

For sparse attention, this is an attempt to change the rules from the ground up.

Oasis Capital: Returning to another emphasis in the paper: you repeatedly mention NSA's "training-aware" design. Could you share the thinking behind your emphasis on "training"?

Dr. Yuan: This issue has indeed been severely underestimated in the past.

Most work in recent years applied attention sparsity only at inference time; once models returned to training, especially pretraining, sparse structures essentially couldn't function.

This is because training and inference differ fundamentally in attention behavior. Training-stage attention needs to learn, and learning requires "gradient signals." If you prune attention to be very sparse from the start, it has no way to know which tokens are important: it lacks space to explore. In other words, it's forced to be sparse before it's learned how to be sparse.

So NSA invested substantial design in "how to let the model learn sparsity."

First, we have the compression branch provide gradient guidance in a stable way. It doesn't directly replace attention, but provides coarse-grained supervision about spatial structure, letting the selection branch gradually learn token-level sparse behavior.

Second, we add sliding window as a consistently reliable local channel. It ensures the model doesn't suffer learning breakdown from sparsity in early training. Without this channel, the model couldn't even learn natural language's own short-range associative structures.

The key insight here is: sparsity isn't a state, but a structure that can be learned.

If you want a model to operate sparsely, you must give it a stable learning path to gradually transition. Future large model trends keep confirming this. Previously, nearly all compute went to pretraining, but as Agent capabilities, long-sequence reasoning, and reinforcement learning become more important, post-training costs are rising sharply.

This means much future optimization space will come from "how models continue learning efficiently in later training stages." If the architecture itself cannot be sparsified and trained end-to-end, it will obstruct future capability improvements.

What NSA aims to solve is precisely the problem this generation of models must face: how to let attention learn while "dense," then operate efficiently and sparsely once mature, while consistently maintaining capability.

Oasis Capital: We've discussed NSA's design from algorithm and kernel perspectives. But in this work, there's another notable angle: the hardware perspective — that sparse attention's emergence represents not just algorithmic breakthrough, but also a trend in modern GPU architecture itself. Could you expand?

Dr. Yuan: Yes, this actually comes from industry observation. When we exchange with many industry teams, we notice an increasingly prominent phenomenon: modern GPU hardware characteristics are pushing the industry toward sparse architectures.

Take current commercial GPUs: their memory bandwidth is far from keeping pace with memory capacity. Memory read latency is often on the order of tens of microseconds, while memory capacity keeps growing. What does this mean?

It means if your model design wastes both bandwidth and memory, the number of tokens it can process per second during decoding will be severely capped. Imagine: in certain international benchmarks, if a model can only generate a few dozen tokens per minute, it's practically unusable.

So today, with high bandwidth as a scarce resource and memory capacity relatively abundant, "dense, undifferentiated KV read/write" becomes particularly inefficient.

Oasis Capital: This sounds like, without changes to GPU structure, the large model industry will ultimately be forced toward sparse approaches.

Dr. Yuan: You could say that.

You can understand it this way: when memory capacity far exceeds available bandwidth, the only way to keep a model's TPS (tokens per second) from dropping is to make attention access more sparse and more selective.

Otherwise, you fall into a paradox, right? The paradox is:

To not waste memory, you must be dense; to not waste bandwidth, you must be sparse.

But these two are inherently in conflict, and sparse architecture is the only self-consistent path through this contradiction.

Put bluntly: even if you invent an extremely powerful attention model — whether based on Linear, Softmax, or MoE-style structures — as long as it's dense, you'll saturate all memory bandwidth, and TPS will drop to undeployable levels.

At that point, the only viable path for the industry is to construct a sparse architecture, matching the model's attention access density to available hardware bandwidth.

This is essentially the logic we see in NSA: not because sparse is "more elegant," but because it's "more realistic."

In fact, once you begin understanding how GPU architecture affects models, you'll discover that many "seemingly algorithmic innovations" are fundamentally about circumventing hardware constraints. The KV access issue we just discussed is a typical example: under sparse structures, without changing KV access patterns, you can never achieve true acceleration.

Many people mistakenly treat this as a "pure algorithm problem," but it's actually a "bandwidth problem" and a "data layout problem."

Oasis Capital: In some of your recent talks, you've also mentioned that "the importance of the post-training stage is rising rapidly." Could you share more?

Dr. Yuan: This is an increasingly evident industry trend. In past years, our compute allocation for models concentrated mainly on pretraining, but as more complex agents, retrieval-augmented systems (RAG), long-sequence reasoning, RLHF, and various reinforcement reasoning paradigms emerge, post-training stages are consuming ever more compute.

This means if the architecture itself cannot support sparse training, post-training compute costs will grow exponentially — to the point where you simply cannot push models toward more complex, longer-horizon tasks.

So I believe NSA's significance isn't merely as an "attention algorithm improvement," but as providing a new foundational assumption for future large models: models need sparse pathways across their entire training and inference lifecycle.

From pretraining to instruction fine-tuning, from reinforcement learning to preference optimization, from long-horizon task execution to agent internal memory management — all these scenarios share one commonality: they require models to maintain efficiency under long sequences.

If attention density doesn't decrease with capability, the entire future ecosystem gets bottlenecked on bandwidth.

So I believe one future direction is more "post-training architecture" innovation. NSA is an early attempt in this direction, demonstrating a possibility: if the architecture itself can absorb sparsity, then computational efficiency across the entire model lifecycle can be reshaped.

Interviewee and Paper Introduction

This Issue's Interviewee: Dr. Jingyang Yuan

PhD candidate at the School of Computer Science Peking University, Dlib Group at the Institute of Data Science and Engineering. Primary research directions are efficient large language models and sparse attention mechanisms; proposed efficient large model architectures including the Native Sparse Attention model (NSA). Honors include Beijing Outstanding Graduate and Peking University Outstanding Graduate.

Paper link: https://arxiv.org/abs/2502.11089

Thank you to Dr. Yuan for participating in this conversation.

Oasis Capital continues to sincerely invite frontline researchers, developers, and thinkers globally to jointly advance this exploration of attention.

If you are conducting attention-related research — whether in algorithms, systems, cognition, neuroscience, or product design and content construction — and would like to explore and discuss with us, please contact us through the QR code at the end.