Harbin Institute of Technology's Meng Weikang: Giving Attention "Edges" | Attention

Over the past few years, the Transformer architecture has served as the backbone of generative AI, effectively defining the industry's technical trajectory. Its core module, Self-Attention, has become near-ubiquitous in vision and language models thanks to its powerful expressiveness.

Yet for all its impressive performance, the practical problems it creates are hard to ignore — the self-attention mechanism carries quadratic spatial complexity, meaning expensive investments and massive energy consumption. Model training thus devolved into an arms race among resource-rich giants, putting the approach out of reach for smaller teams and enterprises.

Against this backdrop, exploration into balancing efficiency and performance has never stopped in academia and industry alike.

Linear Attention represents a notable attempt in this direction. Its basic idea is to replace the Softmax computation in Self-Attention with kernel functions, thereby reducing complexity. However, while Linear Attention's lower complexity is appealing, it suffers from two critical flaws:

• By abandoning Softmax's exponential scaling, the attention distribution becomes "flat" (high-entropy). This flatness weakens the model's discriminative power, causing it to lose sharpness (Spikiness) in fine details and important features.
• Another core problem with Linear Attention: when using non-negative kernel functions (such as ReLU) to approximate Softmax, all negative information gets "clipped," leaving the model able to see only "positive correlations" while missing "negative correlations" or inhibitory relationships. The attention map thus becomes one-sided, compromising expressiveness.

For this installment of our Attention interview series, Oasis Capital invited Weikang Meng, first author of the paper "PolaFormer: Polarity-aware Linear Attention for Vision Transformers" (ICLR '25) and a PhD student jointly trained by Harbin Institute of Technology and Peng Cheng Laboratory, to discuss how PolaFormer addresses these two problems in Linear Attention and to explore the thinking behind his technical journey.

Full article: approximately 7,000 words, estimated reading time 18 minutes. Enjoy.

In the PolaFormer work, the researchers confronted the two problems in Linear Attention and, targeting its performance on vision tasks, proposed an innovative approach — Polarity-Aware Linear Attention, aiming to achieve both better performance and higher computational efficiency.

Specifically, for the high-entropy problem, the researchers believed it could be addressed by designing a new kernel function to reduce the information entropy of the weight distribution. The underlying theory: if an element-wise mapping function has both first and second derivatives greater than zero (strictly monotonic + convex), it can rescale the q, k (weight matrix) responses to reduce information entropy. Therefore, for the concrete implementation of the kernel function design, this work chose to use a learnable channel-wise power function, with mathematical feasibility proofs provided in the paper.

For the problem of lost negative values, this work proposed that polarity-aware computation could enable all elements of the Q and K matrices (attention weight matrices) to participate equally in similarity computation, achieving the goal of not degrading model expressiveness.

(Images processed using PolaFormer show better results, approaching those of Softmax Attention, achieving the goal of narrowing the expressive gap between Linear Attention and Softmax Attention)

Put simply, PolaFormer's design philosophy is this: since both positive and negative information matter, but we don't want to dramatically increase computational cost, why not process them separately? This is a hybrid strategy that hedges against subtraction operations.

Its basic framework: dual-branch polarity modeling + learnable power function

First, it splits Query and Key vectors into positive and negative parts. Then it designs two parallel branches:

• The upper branch handles "same-sign interactions" (positive-positive, negative-negative)

• The lower branch handles "opposite-sign interactions" (positive-negative, negative-positive)

On the Value vector side, it also splits into two halves, feeding into the two branches respectively. This way each branch can independently process its corresponding relationships. Finally, two learnable matrices G^s and G^o are introduced to weight and adjust the results from both branches, which are then concatenated. Meanwhile, the framework incorporates a learnable channel-wise power function, whose role is to make the attention distribution "sharper" rather than "flat," thereby recovering expressiveness close to that of Softmax.

To validate PolaFormer's effectiveness, this work conducted experiments on multiple tasks: object detection, semantic segmentation, image classification, and the Long Range Arena (LRA) benchmark.

Extensive experiments show that the Polarity-Aware Linear Attention proposed in this work can effectively replace the Self-Attention module in Vision Transformer frameworks, demonstrating significant performance improvements on vision foundation tasks and LRA tasks.

Oasis Capital: Could you briefly introduce your background and how you came to the PolaFormer research topic?

Dr. Meng: I'm currently pursuing my PhD at Harbin Institute of Technology (Shenzhen), advised by Prof. Zheng Zhang. I actually chose this problem from considering the gap between industry and academia. Industry has now largely accepted that large models like Transformer are accurate enough, but when it comes to deployment, companies care more about efficiency — whether it can run on mobile devices or purely on the client side. Models at the scale of tens of billions of parameters, like the LLaMA and Qwen series, are still quite difficult to deploy in resource-constrained scenarios.

So what the academic world used to pursue — "higher accuracy" — has gradually shifted toward "higher efficiency" and "better adaptability." That's a transformation. Large companies have data and compute, so they have more room for trial and error and continuous iteration. But for smaller research teams or startups, you have to fill this gap, find ways to make things more efficient, in order to discover breakthroughs in an already crowded large-model landscape.

Oasis Capital: We understand there are many attention mechanism variants — beyond Linear Attention, there's Sparse Attention and so on. We'd like to understand why you initially chose to optimize Linear Attention specifically. What was the thinking behind this?

Dr. Meng: Linear Attention itself isn't a brand-new architecture; people proposed it around 2020. It has some very stable properties. Sparse Attention, by contrast, has its core problems elsewhere. In terms of expressiveness, for instance, it may randomly drop some similarity information. In our experiments, we also found that as model scale increases, Sparse Attention's performance doesn't maintain well. It might work for small tasks — a shallow network with one or two layers can find a balance between efficiency and performance. But when model scale goes up, the lost information becomes substantial.

Linear Attention is different — it doesn't have this risk of "random loss." What it essentially does is rewrite the similarity computation process using matrix factorization, ensuring reduced complexity without losing global information. So whether processing short text or long sequences, it maintains relatively stable performance.

Oasis: So if I understand correctly, your paper mainly focuses on solving two problems: first, that Linear Attention's distribution is too uniform, not sharp enough; and second, that the non-negativity constraint causes some negative interaction information to be lost. PolaFormer was designed specifically to address these two points, right?

Dr. Meng: Yes, basically. Overall, Softmax has some nice properties for attention weights, but it can't be linearized. What we're trying to do is preserve as many of Softmax's properties as possible.

In the paper, I break the problem into two parts: the first is the non-negativity constraint, which causes Linear Attention to lose negative interactions; the second is the lack of spikiness, leading to overly flat distributions.

Both of these issues are fundamentally related to the design of the kernel function. Softmax's kernel function has certain advantages, but due to computational characteristics, it can't be directly converted into a linear form. So our approach is to decompose Softmax's kernel function and approximate it with a new kernel function that maintains non-negativity while recovering distribution sharpness through a power function, bringing it closer to standard Attention's effect.

Oasis: Could you briefly introduce PolaFormer's overall design?

Dr. Meng: PolaFormer's core is a dual-stream structure. We first split the input Query and Key vectors into positive and negative components, then feed them into two separate branches. One handles same-signed interactions — positive-positive and negative-negative; the other handles opposite-signed interactions — positive-negative and negative-positive.

On the Value side, we split the vector in half along the channel dimension, with each branch operating on half of the Value, so no additional complexity is introduced. Finally, the two branch outputs are concatenated through two learnable polarity-aware matrices.

Oasis: From the diagram, this dual-branch structure looks like two separate attention computations. So we're concerned: how do you ensure that overall computational complexity remains controllable when adding the dual branch? Does the dual branch significantly increase costs in practice?

Dr. Meng: Here's how it works. In implementation, among the Q, K, and V matrices, we split V once along the channel dimension into two halves. This way, during dual-branch computation, each branch only operates on half of V, so the two branches together still cover the complete V.

So overall complexity stays the same — it's just redistributing the original computation into two branches, with only a slight increase in computation.

Oasis: Why was Value (V) chosen to be split along the channel dimension?

Dr. Meng: Actually, because computationally, it can only be split along the channel dimension. If we look at the attention formula, each Key (K) is paired with its corresponding Value (V).

Assume K has dimensions n × d, where n is sequence length and d is feature dimension. Then in Attention, K and V correspond one-to-one along the sequence dimension. In other words, K(i) must match with V(i).

So if we tried to split V along the sequence dimension, K and V would no longer align, and the computation wouldn't work.

Splitting along the channel dimension doesn't have this problem. This lets us divide V in half, sending each portion to a separate branch for same-signed and opposite-signed computation, while maintaining structural soundness. See the formulas in the paper.

Oasis: Next I'd like to ask about function design and selection — I understand this is also a key issue in this work: what to use in place of Softmax.

In the paper, you mathematically proved choosing a learnable channel-wise power function to replace Softmax's exponential function. Could you elaborate: what was your thinking and logic when selecting the function? Theoretically many functions could achieve entropy reduction — why ultimately choose this power function? Did you try other functions?

Dr. Meng: Here's the thing. We proved in the paper that if an element-wise function has both first and second derivatives greater than 0 — strictly increasing and convex — it guarantees that the attention distribution's entropy will decrease.

But there's actually another issue: while all such functions satisfy "derivatives greater than 0," how much greater, and what the function's curvature looks like in different intervals, is hard to define uniformly. And this difference significantly affects how much the distribution is compressed or amplified. We don't really have a definitive answer on this either.

So we ultimately chose the power function because it's very simple and intuitive, and easy to control convexity strength through the exponent p. We also designed this exponent to be channel-wise learnable, so different channels can adaptively adjust their own "sharpness."

In comparison, previous methods like Lightning Attention, or earlier approaches like Linformer and RNS, never considered how to solve the "overly flat distribution" problem from the angle of function selection.

You could say our work was the first to explicitly raise this issue.

Oasis: This is also one of the main contributions of this paper?

Dr. Meng: Yes, you could say it's one of our more core contributions. We ultimately settled on the power function because when tuning, you can more directly control the attention distribution's spikiness through the exponent size — making it more concentrated or smoother. You can see this effect quite intuitively in the experiments.

Of course, this is just a first step. We haven't fully explored, for instance, how sharp is sharp enough, or what range of entropy reduction is optimal — these don't have definite answers yet. In other words, we've just proposed a direction: proving that power functions are effective and can strike a balance between expressiveness and efficiency. As for more fine-grained parameter selection, that needs further exploration in the future.

Oasis: I think this is also a very interesting point. Because from this angle, we can better understand the "potential" research directions mentioned in the paper.

Dr. Meng: Yes. Much previous work just said "we propose a kernel function" without clearly specifying what properties this kernel function should satisfy.

Our work provides a relatively systematic approach: if you're designing a new kernel function in linear attention, what minimum conditions should it satisfy to ensure the attention distribution is both efficient and sufficiently expressive.

So I think what's important isn't using the power function itself, but proposing a set of generalizable design principles. This means future researchers can build on this to try more types of functions — as long as they satisfy these conditions, they should theoretically achieve similar effects, possibly even better ones.

Oasis: Could you help us summarize the framework for kernel function selection? In this framework, there's actually freedom in function choice, but also some constraints. Could you summarize these constraints again?

Dr. Meng: Sure, I think there are roughly three core aspects.

First, the non-negative constraint. In attention computation, we must guarantee weights are non-negative — this is the most basic condition.

Second, spikiness. In our paper, we propose addressing this through dual streams, making sure the attention distribution doesn't become too flat.

Third, full-parameter participation. This is actually quite crucial too. In standard Attention, every element participates in computation, but many Linear Attention methods, to satisfy the non-negative constraint, simply truncate all negative values. The result is that potentially half or more of the information in the attention distribution gets discarded — roughly 3/4 of the content is missed.

Oasis: I want to confirm: does full-parameter participation count as an independent constraint? Or is it actually solving a problem caused by the non-negative constraint?

Dr. Meng: I think it should count as a separate constraint. Because relying solely on the "non-negative constraint" is hard to solve all problems. Each input has a different distribution — you can't guarantee one dimension is always positive and another always negative. In this uncontrollable situation, much previous work took the approach of "just throw away all the negative parts, keep only the positive ones," like using ReLU to zero out all negatives. While this guarantees non-negativity, the problem is it's too crude: some meaningful negative similarities get mapped to 0, causing severe information loss.

For example, if one dimension of a query and key vector is -100, and the other's is -99, both have large absolute values and may be important for the model's expressiveness. But once passed through ReLU, both become 0, and that distinction disappears entirely. This leaves the model without the ability to differentiate during learning.

Oasis: In your paper, you propose a learnable polarity-aware mixing mechanism. Could you explain specifically how these two streams demonstrate complementarity? Have you encountered situations where both streams learn similar patterns and lose that complementary relationship?

Dr. Meng: This question can be answered directly by looking at Figure 3 on page five of the paper.

You'll notice that the weight distributions of (G^s) and (G^o) do overlap in parts, but their overall patterns differ. (G^s) (same-signed stream) is more concentrated, mostly falling between 2.5 and 5.0 with a relatively uniform distribution; while (G^o) (opposite-signed stream) is more dispersed, indicating functional complementarity between them.

The original intention behind this mechanism was to capture both same-signed and opposite-signed interactions. As you can see from the formulas, these two parts were originally separate, and we hoped that through learning G^s and G^o, they could be effectively combined. But guaranteeing this complementarity isn't easy, especially during large-scale training — you can't fully ensure the two are always strictly negatively correlated.

That's why we introduced this "mixing gating" mechanism. Its advantage is that even when their distributions overlap in certain cases, the model can automatically balance them through learnable parameters. Experimental results show that this design does lead to noticeable performance improvements.

Oasis: PolaFormer applies element-wise scaling to the results of same-signed and opposite-signed interactions, then lets the model learn the weight balance between these two parts itself. In other words, we hope that through learnable G, the model automatically learns a negative correlation relationship between same-signed and opposite-signed interactions.

We saw performance improvements in experiments, but how do you prove the model actually learned this "negative correlation"?

Dr. Meng: Right, this question again comes back to Figure 3, because Figure 3 is a visualization showing the weight distributions of G^s and G^o. This visualization is based on PCA, since the matrix itself is high-dimensional and not intuitive to view directly.

Looking at the results in the image, the learning of G is generally reasonable: they did learn a complementary relationship. But you might notice the blue dots look a bit "messy" — this is because the model has to satisfy two conditions simultaneously: first, the non-negativity constraint; second, negative correlation.

Strictly speaking, these two conditions are contradictory. If you fully guarantee non-negativity, you can't maintain negative correlation.

So our final conclusion is: What G learns is actually a trade-off, approximating the negative correlation between the two branches as much as possible while ensuring non-negativity.

Oasis: In your paper you discuss two rank-raising methods: depthwise convolution (DWC) and deformable convolution (DCN). You ultimately chose the former. Could you elaborate on why you made this choice?

Dr. Meng: Actually, the choice of convolution isn't the core innovation of our work — it's more of an engineering consideration. We found in our research that the similarity matrix in Linear Attention tends to suffer from low-rank issues. For instance, when sequence length is large but feature dimension d is relatively small, the matrix from multiplying Q and K will have insufficient rank. This makes it difficult for the model to learn complex relationships.

So we needed a method to "raise the rank" and make the matrix rank higher. Convolution happens to introduce some local information, equivalent to adding extra independent features that push the matrix closer to full-rank, making it easier for the model to learn.

In terms of selection, we tried both DCN and DWC. DCN is theoretically more flexible, but implementation and tuning are more complex; while DWC achieves comparable results with simpler computation and higher efficiency. So we ultimately chose DWC.

But this part isn't our core contribution — it mostly draws on existing work to address the degradation in V matrix learning capacity caused by low rank.

Oasis: Looking ahead, what do you think should be the core breakthrough for Linear Attention's next step?

Dr. Meng: I believe attention mechanisms already have strong expressive power; the core problem lies in efficiency. Especially in edge deployment, compute-constrained scenarios, and large model training, the overhead is too high. In other words, the next breakthrough is still how to make Attention more "lightweight" while preserving expressiveness.

Oasis: What are your thoughts on algorithm-hardware co-design?

Dr. Meng: Hardware constraints definitely need to be considered now. For example, in actual training or deployment, long-sequence models often don't fit on a single GPU and need to be split across multiple cards for parallel processing, which brings communication overhead, gradient synchronization, and memory limitations. So future Attention optimization must co-design with hardware to truly solve deployment bottlenecks.

Our research into Linear Attention is also aimed at maintaining efficiency under these constrained conditions. In applications like video processing and large language models, its potential is significant. But some industry conventions, like MiniMax's Flash Attention, actually use a hybrid approach. Their method seems to be 7 layers of lightening attention plus 1 layer of standard Attention, combined in a 7:1 ratio. This sacrifices some accuracy but significantly reduces computational and time overhead.

Oasis: At the industry level, what changes or benefits do you think PolaFormer can bring to practical applications?

Dr. Meng: Currently most of our experiments are still fairly traditional — image classification, detection, segmentation, and other CV tasks. But what I find more promising for the future is its application in long-sequence and high-resolution scenarios, such as video processing, large language models (LLaMA series, etc.), and inference and training for high-resolution generative models.

We've done some testing on LLaMA and found noticeable efficiency improvements in long-context settings (4k+), but due to resource and data constraints, we haven't fully completed this yet. My inclination is to treat this as a focus for follow-up work, expanding it into a more general model.

From an industry perspective, tasks like long-sequence text and high-resolution image generation are extremely time-consuming, both in training and inference. Linear Attention can provide a more lightweight solution while guaranteeing performance, and I believe its prospects are brighter than standard attention mechanisms. It will reduce hardware requirements for Transformer-based models and help large Transformer models reach practical deployment.

Additional Information:

Meng Weikang is the first author of the paper PolaFormer: Polarity-aware Linear Attention for Vision Transformers. He is currently a joint PhD student between Harbin Institute of Technology and Peng Cheng Laboratory, with a bachelor's degree from Harbin Institute of Technology. His research focuses on efficient training and inference algorithms for large-scale foundation models.

Meng Weikang's advisor and corresponding author of the paper is Professor Zhang Zheng, currently a tenured professor and doctoral supervisor at Harbin Institute of Technology, a Ministry of Education "Young Changjiang Scholar," a Guangdong Special Support Program "Young Pearl River Scholar," and a Shenzhen Outstanding Young Talent. He has long been dedicated to efficient multimodal machine learning, with particular focus on frontier exploration of efficient and trustworthy multimodal large models.

Paper link: https://arxiv.org/pdf/2501.15061

GitHub link:

https://github.com/ZacharyMeng/PolaFormer

Thank you to Dr. Meng for participating in this conversation.

Oasis continues to sincerely invite frontline researchers, developers, and thinkers worldwide to jointly advance this exploration of attention.

If you are working on attention-related research — whether in algorithms, systems, cognition, neuroscience, or product design and content construction — and would like to join us in discussion and dialogue, please contact us through the QR code at the end of this article!