Tsinghua's Tianchen Zhao: When Models Learn to Only Look at What Matters | Attention

In the Oasis Attention series, one underlying question we've kept circling back to is this: as input sequence lengths for generation tasks continue to grow, token sequences frequently reach scales of 10K–100K, especially in visual settings. In this regime, the Attention operation, with its quadratic complexity relative to sequence length, becomes one of the primary performance bottlenecks hindering both model inference and training.

Numerous outstanding papers and their research teams have shared their approaches to this problem with Oasis, each from a different research perspective.

Research on Attention efficiency has primarily developed along two technical paths. One family of methods reduces the number of attention connections through Sparse Attention, retaining only a subset of interactions deemed important — a topic we've covered in depth in past installments of the Attention series.

The paper we're discussing today — PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models (hereinafter "PARO") — shares the sparse path's concerns while also exploring the other route: replacing high-precision floating-point computation with low-bit integer operations such as INT8 or INT4 through Attention Quantization.

These two approaches have achieved some success in language models and certain visual tasks. However, in visual generation tasks, they often degrade noticeably under low-density (e.g., <50%) and low-bit (especially pure INT8/INT4) settings.

PARO attributes this to the "complex and irregular" nature of visual attention patterns: attention weights exhibit diverse spatial forms, dispersed distributions, and vary across timesteps, conditions, and attention heads. This makes sparse masks difficult to design in a general, structured form, while also increasing outlier values within quantization groups — amplifying quantization error.

For this installment, we've invited Tianchen Zhao, first author of PARO and a PhD student in the Energy-Efficient Computing Lab at Tsinghua University's Department of Electronic Engineering, to share his and his team's research perspective and thinking with Oasis.

Rather than optimizing around "how to design masks/groupings for complex patterns" like existing approaches, the paper takes a different tack: changing how input data is arranged during Attention computation. Through Token Reorder, it reorganizes scattered, chaotic attention patterns into unified, regular block structures, making subsequent sparsification and quantization structurally easier to implement. This reordering scheme is defined as Pattern-Aware ReOrdering (PARO).

The paper analyzes "locality" in visual tasks (Locality: in most generation stages, a single token's effective attention weights concentrate primarily on a small subset of tokens spatially or temporally adjacent to it. This adjacency may correspond to spatial neighborhoods within the same frame, or temporal relationships across adjacent frames, with specific forms varying according to model architecture, timestep, and conditional input.), noting that diverse attention patterns in 3D space mainly correspond to local aggregation relationships — yet under the default token flattening order, these local relationships are stretched apart in the 1D sequence, manifesting as dispersed structures such as multiple diagonals, stripes, or sparse blocks.

PARO's approach is to select appropriate token dimensional permutations for different attention heads (choosing among different permutations of [F,H,W]), transforming local aggregation that originally occurred on non-contiguous dimensions into aggregation along memory-contiguous dimensions — thereby concentrating attention structures more tightly within blocks.

Building on PARO's attention restructuring technique, the authors further propose PAROAttention: a unified block-wise attention pattern enabling synergistic sparsification and quantization. The reordered structure makes structured sparse masks easier to form, yielding actual hardware benefits under FlashAttention's block-based computation paradigm; meanwhile, with attention distributions more concentrated within blocks, outlier problems in quantization groups are alleviated, making INT8/INT4 more viable.

Experimental results show that this method achieves significant acceleration in video and image generation tasks while largely preserving generation quality, and supports more aggressive low-bit settings.

Key Contributions

For the Attention efficiency problem, PARO provides a preprocessing approach centered on token reordering, unifying diverse attention patterns into hardware-friendly block structures — thereby reducing the difficulty of sparse mask design and improving outlier issues under quantization grouping.

PAROAttention implements synergistic sparsification and quantization on this structure, supporting low-bit settings such as INT8/INT4 while maintaining generation quality, and achieving 1.9–2.7× end-to-end speedup on models including CogVideoX and Flux. The method offers certain generality and deployment potential as a systematic path for attention compression and acceleration in visual generation models.

Below is the full transcript of Oasis's in-depth conversation with Dr. Tianchen Zhao — approximately 20 minutes of reading.

Enjoy

Oasis: Hello Dr. Zhao, thank you so much for joining us to share your thoughts on this work. Before we dive into the research itself, could you briefly introduce your background and research direction? And how did you originally choose to enter this field?

Dr. Tianchen Zhao: Sure, no problem. My name is Tianchen Zhao, and I'm currently a PhD student in the Department of Electronic Engineering at Tsinghua University. I took the direct-from-master's path, starting my PhD after completing my master's, and I have about one year left until graduation.

I joined Professor Yu Wang's research group at Tsinghua's Department of Electronic Engineering during my master's. Our group focuses broadly on energy-efficient computing — the core goal is to enable current mainstream, computationally intensive deep learning models to run more efficiently on hardware, that is, to reduce computation and energy consumption without sacrificing performance as much as possible. This direction covers a fairly wide range of layers, from algorithms to systems and the co-design between them.

Within the group, I personally lean more toward algorithms, mainly working on efficient deep learning algorithm design for specific application scenarios. If I had to put it in a nutshell, it's about building efficient models.

Around 2022 to 2023, large models began developing rapidly, and the field's research focus shifted quite noticeably. The group's main research thread followed large language models, while I — having some computer vision background — focused on another promising technical direction: visual generation.

At earlier stages, people typically viewed text generation models and image generation models as two relatively independent technical paths. But later, you could see these two paths gradually converging. On one hand, from a multimodal perspective, more and more work began attempting to integrate perception, understanding, and decision-making within a single model; on the other hand, generation tasks themselves were no longer limited to single-modality output, but progressively expanded toward multimodal generation.

So what had originally seemed like divergent paths for text generation models and visual generation models began converging at the level of overall technical direction, gradually moving toward a more unified approach to multimodal modeling.

Oasis Capital: Let's come back to the paper itself. When you first set out to write it, what was the core problem you wanted to solve? What was your entry point?

Tianchen: The thing I most wanted to address, at its heart, was locality in vision. I've always felt that locality is an absolutely critical property in vision — one that's been validated again and again.

If you look back at the history of computer vision, this is abundantly clear. Around 2012, for instance, deep neural networks really "took off" largely because convolutional neural networks achieved breakthrough results on vision tasks. And CNNs worked precisely because they introduced a very strong inductive bias through local connectivity and parameter sharing.

From another angle, you could also frame this as a question of "optimization space."

If you compare a fully connected layer against a convolutional layer, in theory the fully connected layer has greater expressive power — its parameter space can subsume the convolutional layer as a special case. Yet in practice, fully connected layers are actually harder to optimize well. It's not that they aren't powerful enough; it's that their optimization space is too vast to be effectively exploited under constraints of limited data and optimization algorithms.

Convolution introduces a locality assumption that constrains what would otherwise be an enormous search space into a more manageable subspace, making the model easier to converge and more likely to work on real tasks. I've always tended to understand this kind of inductive bias as: before we even begin modeling, we deliberately introduce certain "prior assumptions" that make the problem itself more tractable.

The autoregressive modeling framework that made large language models successful, for instance, can also be seen as an inductive bias for "sequential data" — with natural language as the canonical example.

From this perspective, locality isn't merely a design principle distilled from historical experience; it's also a structural property implicit in vision tasks themselves. The question isn't whether locality exists, but how to leverage it well within new model architectures.

This was really a continuation of my earlier work. Going back to around 2022, we had experimented with some Attention optimization ideas in settings like 3D point clouds, but the application scenarios and hardware constraints at the time weren't mature enough for those methods to reach a "must-have" stage.

So later I went looking for a scenario with genuine hardware pressure — where Attention's computational overhead would be large enough that, without optimization, it would become systemically unacceptable. Because Attention's computational complexity is O(N²), this problem truly surfaces as input sequences grow longer.

This condition gradually came into place for video generation models around mid-2024. Some 3D full-attention video generation models began to emerge, token counts grew significantly, and Attention's computational costs were materially amplified. Before this, many approaches had adopted relatively conservative strategies — for example, decoupling the temporal and spatial dimensions and using factorized methods to avoid true global Attention — so Attention's proportion of overall computation wasn't particularly high.

But when models began deploying Attention simultaneously across both temporal and spatial dimensions, and total token counts entered a new order of magnitude, Attention costs truly became the primary bottleneck. Against this backdrop, we felt it was the right moment to systematically consider Attention optimization, rather than remaining at the conceptual level.

Oasis Capital: Let me briefly summarize — are you saying that you wanted to reintroduce locality from vision into the Attention structure, and to do so in a real application scenario with genuine hardware and computational pressure?

Tianchen: Yes, that's essentially it.

Oasis Capital: Let's move into the technical solution itself. The Pattern-aware token rearrangement mechanism you proposed — what's the overall technical architecture? Could you walk through how you approached this?

Tianchen: Sure, I'll try to explain this top-down. The core of this problem really comes down to the specific research approach, particularly how the "optimal permutation" is determined.

Oasis Capital: My understanding is that you started with 3D tokens — say, frames, spatial height, spatial width — but in the model these typically get flattened into a 1D sequence, which disrupts the original adjacency relationships between tokens. You wanted to reconstruct a more suitable permutation so that subsequent sparsification or quantization could work more effectively. What I'd specifically like to understand is: how did you determine this "optimal permutation"?

Tianchen: Actually, we were just discussing this with reviewers a couple days ago.

Our current design is actually fairly simple. We didn't construct a massive search space; we only select from a very small candidate set. Specifically, for the three dimensions in video tokens — temporal and two spatial dimensions — we only considered the full permutations of these three dimensions, meaning six (3 factorial) possible arrangements in total.

The reason for this is primarily based on patterns we observed when analyzing Attention maps. We printed out large numbers of Attention maps and found a fairly interesting regularity: during training, different Attention heads tend to learn aggregation relationships along one particular dimension, or along the direction of two particular dimensions. In other words, each head typically develops a preference for forming relatively pronounced local structures in one particular direction.

Based on this observation, our approach is: among these three possible dimensional directions, we find the most "local" direction for each head, then sort tokens according to that direction. After doing this, relationships that were locally aggregated in 3D space become easier to organize into relatively regular block-like structures in the 1D sequence.

I'm not sure if that was clear enough.

Oasis Capital: Yes, very clear. Let me follow up: is this "optimal permutation" fixed across different permutations, or does it need to be recomputed each time?

Tianchen: That's a crucial question. Our solution is model-dependent — that is, it's tied to specific model weights.

But for a given model, once the permutation is determined, it's fixed. This is intuitive because the model weights have learned "a certain tendency toward locality," which is a "characteristic learned by the model" that's "encoded" in the weights.

In other words, for the same model, regardless of different timesteps, different input noise, or different text conditions, we use the same permutation — there's no need to repeatedly recompute or adjust it during inference.

Oasis Capital: Let's get more specific about your actual operational process. I see that your solution first determines the optimal permutation offline, then executes it in the online phase, using this approach to improve hardware efficiency.

What I'm particularly interested in is: how is this offline determination step integrated into your overall pipeline? Does it require switching back and forth during runtime, or is it a single-threaded, seamless process?

Tianchen: This can be understood as two stages. In the first stage, before inference or generation actually begins, we complete all the steps that need to be done offline based on some representative data — essentially calibrating everything that needs "calibration" ahead of time.

After the offline stage concludes, we already know how the token order should be permuted for every Attention head, or every class of head. This result can be understood as an index, or a set of mapping relationships.

In the online stage, when the model is actually running, it doesn't need to do any complex searching or judgment. It simply follows this already-determined index to execute the corresponding permutation, then proceeds directly into the subsequent computation flow. So from an overall perspective, these two stages are fairly decoupled: offline handles the complex work in advance, while online is kept as streamlined as possible.

This is also a fairly clear distinction between our approach and some contemporaneous work. Many methods, in pursuit of stronger dynamic adaptability, place quite a few steps in the online phase, hoping the model can dynamically adjust based on different input conditions. But in our analysis, we found that this on one hand introduces rather complex online computation and control logic, and on the other hand doesn't necessarily yield optimal acceleration.

We observed that certain structural choices can actually generalize across different timesteps and noise conditions. For instance, token permutation order doesn't need to change with every input variation. So we chose to move these steps to the offline phase and complete them once. This way, the online phase basically incurs no additional computational overhead — it only needs to execute the already-determined acceleration path.

Oasis Capital: So this is essentially a static process, correct?

Tianchen: Yes, it's static. This is actually something we intentionally wanted to emphasize.

There's often an intuitive feeling that dynamic methods sound more algorithmically sophisticated, but after fairly systematic analysis, we found that dynamic strategies tend to bring extra overhead, with a slight penalty to hardware efficiency. Additionally, from an algorithmic effectiveness standpoint, because online solutions can only determine sparsity based on the Attention map before Softmax, we found that in many cases they're actually less accurate than approaches that leverage the offline Attention map after Softmax. Taking everything into account, we chose the static solution to achieve a better performance-efficiency trade-off.

Oasis: I wanted to ask one more question about generalizability. Beyond the generative architectures discussed in your paper—say, in other image or video generation models—could this approach based on reordering and parallel execution also be applied?

Dr. Tianchen: That's actually a question I already answered for the reviewers.

From our current understanding, this method does have a certain degree of generalizability. The main reason is that our starting point wasn't any specific model architecture, but rather two fairly universal facts: first, that visual generation tasks inherently exhibit locality; and second, that when Transformer architectures process visual data, they typically need to flatten two-dimensional or three-dimensional structures into one-dimensional sequences, and this Reshape or Flatten operation disrupts the original data adjacency relationships.

From this perspective, as long as it's an Attention-based model processing visual data—whether image generation or video generation—it will be affected by this issue to varying degrees. Therefore, as long as the model contains a similar Attention computation structure, our method is in principle applicable.

Beyond generation tasks, we've actually done supplementary experiments on some understanding tasks. For instance, in visual understanding scenarios, some efficient models attempt to use Diffusion Models to replace traditional language models or other structures. We ran experiments on one such method (Llada-v), which also has a vision-related variant. The results showed that similar phenomena can be observed in these models as well, and applying our solution yields fairly consistent gains.

Oasis: I noticed your paper includes quite a lot of experiments and evaluation results. Could you briefly summarize which models you mainly tested on, and what the overall performance looks like?

Dr. Tianchen: We started this work relatively early, around the end of last year, so the first batch of experiments mainly focused on video generation models. At that time, our primary experimental subject was models like CogVideo, which is also the core experimental scenario in our paper.

Later, as larger-scale and more community-recognized open-source models emerged, we added some supplementary experiments. For example, we also validated on Alibaba's Wan model. CogVideo served as the main experimental model, while Wan was more of a supplementary experiment to verify how the method performs under different model architectures.

Looking at the overall results, in typical video generation scenarios, our conclusion is: after applying sparsity and quantization, it's difficult to observe any noticeable quality difference from a human subjective standpoint. That is, there is no visually perceptible degradation in generation quality.

For sparsity, under relatively stable settings, we can roughly reduce the effective computation density of Attention to about 20% on average—equivalent to cutting the original computation to roughly one-fifth. This setting maintains fairly stable results in most video generation scenarios.

For quantization, compared to previous work, our solution can push the entire Attention computation pipeline to INT8. For some models and scenarios, it can be pushed further to INT4. However, it should be noted that for video generation models, INT4 is already a relatively extreme setting, and in some cases instability or slight flickering begins to appear.

We also conducted experiments on image generation models. Taking certain image models as an example, they actually adapt better to quantization—even at INT4 settings there are basically no obvious issues; but in terms of sparsity, the compressible space is relatively limited, and a sparsity rate around 50% tends to be safer.

Overall, whether for video generation or image generation, the trend in efficiency improvement is fairly consistent—it's just that in the two dimensions of sparsity and quantization, different models have different "comfort zones."

Oasis: Finally, I'd like to ask about your vision for the future development of this solution. Which domains do you think it could be applied to? And what kind of impact might it have?

Dr. Tianchen: If we're talking about this solution itself, I think it comes back to the starting point we've been emphasizing: locality in visual data perception. Locality itself is fairly universal—it doesn't just appear in one specific task. As long as you're modeling images or videos, whether for generation or more broadly for perception and representation learning, you'll likely encounter the problem of local structure.

Compared to language, images are actually a quite different modality. Because language itself is a symbolic, highly abstract system. Images are a more intuitive modality, closer to the sensory level.

If we view humans as an intelligent system that "compresses and processes data," the data bandwidth that our eyes and brain encounter is actually quite different. Video, for instance, is a very high-bandwidth data source; we need to first perceive and model it into "concepts" that can be apprehended through symbolic language. I personally feel that these "low-level" sensory-perceptual representations closer to raw sensation, compared to the "high-level" symbolic concepts of language, involve greater data bandwidth and even an explicit "compression mapping" process. And I tend to believe that "locality" is one of the most critical characteristics of the visual modality—an important inductive bias in the "symbolization" of visual information (analogous to Tokenization in the current large model context), and even in the interactive reasoning between symbols (analogous to how current large models use Transformer Attention).

Specifically, the current solution is still a very conservative and concrete one—merely a post-training compression scheme for existing models. We actually think there is considerable room for extending similar ideas in terms of technical implementation. For example, modifying the Attention mechanism itself so it can be used during mid-training or even pre-training, allowing the model to naturally learn to "exploit locality" rather than post-hoc "mining the locality patterns the model has already discovered."

We hope that with further exploration, this solution and the underlying thinking might influence or even reshape the design of model architectures and basic feature extraction operators for visual perception.

Overall, the core idea of this paper—"adjusting distributions to better exploit the local characteristics of data"—is a problem tied to the visual modality. If we further consider cross-modal situations, such as the relationship between text and images, things become quite different. Text-image Cross Attention, in terms of task form and attention patterns, is inherently not identical to intra-image Attention. So in its current form, our method may not be directly applicable to text-image relationship modeling.

Intuitively, text-image Cross Attention is more like doing an alignment or mapping subtask, rather than pure local aggregation. Although they coexist in the same model, these two types of Attention differ in pattern. My personal understanding is that they are themselves two somewhat different subtasks.

Oasis: But from an application perspective, in many practical systems, scenes are often driven by text generating images or videos. In such settings, could this method still potentially play a role?

Dr. Tianchen: In principle, yes, but it depends on how the actual system is implemented. As I just mentioned, text-image Attention does exist in current models, but its proportion in overall computation is actually not large.

In currently typical generation models, text token lengths are usually not particularly long—perhaps on the order of a few hundred; whereas images or videos, after being unfolded into tokens, often reach tens of thousands or even hundreds of thousands. There is already an order-of-magnitude gap in computation between these two parts.

So in the technical solutions we're currently focused on, the main optimization target is still intra-image or intra-video Attention, because this is the part with the highest computational proportion and the most likely bottleneck. Text-image Attention, while important for the task, is not currently the main problem from a computational cost perspective. As for the future, if model architectures change or cross-modal computation scales significantly, that may need to be reevaluated. But at this stage, we prefer to concentrate our efforts on structural optimization within images and videos.

Oasis: Alright, to wrap up, let's return to our theme: what role do you think Attention mechanisms play in your field of research?

Dr. Tianchen: If I first place myself in the perspective of "doing efficient computing," this question is actually quite straightforward.

First, Transformer models, including their Attention operators, currently show a very strong trend toward unifying various tasks as the "sole model architecture." For Attention, as long as model sequence lengths grow to a certain scale, Attention will almost certainly become the primary bottleneck in both training and inference. So from an engineering perspective, it's an unavoidable module that must be optimized. The core problem with Attention itself lies in its computational complexity of O(N²). It's a global aggregation method—that is, every token in the sequence establishes relationships with all other tokens. At smaller scales, this is acceptable; but as token counts continue to grow, this global modeling approach becomes a major obstacle to algorithmic scaling.

Precisely because of this, a tremendous amount of work over the past few years has attempted to address Attention's efficiency problem. There is widespread consensus that global Attention is redundant in many cases, and that it is not a mechanism naturally suited to scaling. This has made "how to make Attention less global" a very open but extremely important problem.

Many different technical approaches have already emerged around this problem. For example, sparse Attention (such as NSA, MoBA), linear Attention (such as DeltaNet, SANA, Qwen-Linear), or other "sub-quadratic complexity" Attention variants (such as Mamba) — all of these can be seen as simplified intermediate forms of the full global Attention, adding some kind of "inductive bias" to optimize within a limited space rather than across a broad global space.

At the same time, many people are skeptical of these Attention improvements, believing that the weak inductive bias of global Attention is precisely what enables scaling up data and compute, and is even a necessary condition for Transformers. They argue that some efficient Attention variants attempt to use a "limited and fixed" memory to store and represent "infinite" sequence data, making it difficult to achieve good results from the outset.

Overall, how to design an Attention mechanism that is both efficient and maintains modeling capability is itself a major Open Problem. It is also an unavoidable and critically important problem for building foundation models, from both algorithmic and systems perspectives.

Oasis Capital: Indeed, there's a tremendous amount of research on Attention efficiency right now. But if we broaden the perspective further, how do you see the role of Attention in longer-term, even cross-disciplinary applications?

Dr. Tianchen: I think before discussing this, we may need to first define our terms. When we say Attention here, are we referring to the specific QKV computational form, or a more abstract sense of feature aggregation? Frankly, from what I've read and know, we haven't yet fully understood "why Attention in Transformers works." If we had, we wouldn't need so much exploration into Efficient Attention.

I think if we stand at a higher, cross-disciplinary level exploring ideas, in this context Attention will likely no longer be that specific computational graph.

We can first consider the meaning and future of Attention in the deep learning context. If I were to find an answer to "why Attention works" right now, I would actually say it's precisely "smaller inductive bias" — which may seem contradictory to our work's attempt to add inductive bias to Attention. Let me explain.

Attention's success is inseparable from the current era of large-scale pretraining as the mainstream. Before the LLM era, the Transformer architecture gained widespread recognition in NLP precisely because it enabled larger-scale parameter models and larger-scale pretraining (compared to other specialized models at the time). "Introducing less human prior and relying on larger-scale models and data" is what the success of deep learning has taught us. Attention, as a "general sequence data modeling" approach (compared to local convolution, which may correspond more to visual data), doesn't incorporate much human prior, aligning with the "Scale Up" mainstream of the era. But this still doesn't answer "how the current QKV Attention approach works" — I think this is a rather fundamental Open Problem. If people can more clearly understand the essence of this matter in future practice, that would be a prerequisite condition I'm optimistic about for Attention mechanism's future development and improvement. Still, I believe Attention has room for progress regardless of the direction.

If we elevate further and consider cross-disciplinary thinking — from neuroscience, psychology, and even philosophical perspectives on Attention's role — I think we can actually provide more analysis and interpretation for Attention mechanism's success.

I may need to offer a disclaimer first: I'm personally a science student, and my engagement with humanities and philosophical thought can only be described as superficial. The personal views below certainly contain many very imprecise and even misguided elements, but I'll share them nonetheless.

Attention essentially appears as the most core "feature extraction operator" in current neural network models — that is, in "how to integrate and distill characteristics from data." So if we consider large models as analogs to human cognition to some degree, then the Attention mechanism becomes the core mechanism of so-called "silicon-based intelligence," a crucial role. Analogous to our brains, it may be a "fundamental mechanism" equivalent in status to "the aggregation of large amounts of electrical pulse signals between synapses." On this point, the view I want to express remains that "Attention's success stems from it being the mechanism most suited to contemporary deep learning, and its thinking also represents a microcosm of deep learning mechanisms."

What thinking? "Building cognition from observed data associations/correlations" — I think this is actually the "comeback" of empiricism. From the earliest AlexNet replacing expert systems, a key characteristic of deep learning was "letting models learn patterns themselves from large-scale data" rather than based on pre-constructed "expert knowledge systems." This actually forms an interesting duality with empiricism and rationalism. Empiricism holds that the source of knowledge is experience, or sensory perception; its diametrical opposite, rationalism, holds that knowledge derives from innate ideas or rational deduction.

Deep learning abandoned the "rational framework" constructed from "human priors" of expert systems, instead simply inducing and summarizing patterns and knowledge from the correlations in "observed data." Its optimization objective is also maximum a posteriori probability — to draw an imperfect analogy, the model is predicting "what is most likely to happen" because it has seen similar situations in training data. This point has actually been quite questioned and criticized, with some feeling that this correlation-based (Correlation) modeling cannot truly learn "causality" (Causality). But extensive experimental results show it works. Corresponding to neuroscience and human intellectual history, similar viewpoints and "route disputes" also exist.

From the Enlightenment onward, the debate between empiricism and rationalism raged until Kant reconciled the two, though mainstream thought tended more toward rationalism, holding that human thought is built on rational reasoning — this is what our textbooks mainly tell us. But actually, the success of deep learning has some impact on the question of "how humans think": is human thought process "doing rigorous reasoning based on existing prior concepts," or is it also a "maximum a posteriori probability" that makes our cognition conform to our observations?

Some neuroscience perspectives and findings may make this open question easier to accept. I previously came across an interesting neuroscience sub-school called Predictive Coding. They argue that our brains don't "generate consciousness," but rather construct a "most likely self-consistent reasonable explanation" for the external world we observe, making these things "make sense." They also mention some interesting real-world experimental results. For instance, some studies found that the muscle electrical signals produced by body movement actually occur earlier than the brain's electrical pulses, so it's not that the brain is "controlling the body," but rather that the brain is "predicting the movement that is about to occur."

Returning to Attention: the Query-Key-Value design of Attention, from its earliest Dictionary Learning perspective interpretation, holds that Value is a value in a dictionary representing learned information and knowledge, while Query and Key will do an inner product (computing correlation) to produce an Attention Map ([N,N]) that models the "relative relationship" between every input Token and all other Tokens. This relative relationship is then used to represent how to "query and combine knowledge," letting the Attention Map do another inner product with V. Such an architecture indeed has much similarity with the empiricist description of how human knowledge is produced — our thinking process uses sensory perceptual inputs to "index and match" historical experience.

I would say Attention's success can serve as an excellent abstract model of "building correlations between things and generating experience," to serve as a thought and methodology that inversely inspires our understanding of "human cognition."

But returning to practical matters, from the foundation model building perspective, given the current model ecosystem, infrastructure, toolchain, and the enormous sunk costs already invested, in mainstream application scenarios, I personally am relatively optimistic that Attention will remain in its current form as the most critical feature extraction operator for a considerable period ahead.

Other efficient solutions, if they want to "replace" Attention on mainstream tasks, will need to wait until the Attention mechanism itself encounters first-principle problems it cannot solve. Current efficiency issues still fall within the realm of solvability through approaches like local Attention or other levels of compression.

Oasis Capital: So your judgment is that the overall Transformer architecture won't be easily replaced, but the specific implementation form of Attention will still evolve?

Dr. Tianchen: Yes, that's roughly my view. Transformer as a unified modeling framework, I think it will exist for the long term. But Attention as one specific component within it will likely continuously evolve. In the short term, this will be due to hardware system constraints requiring some minor adjustments (such as sparsity, low-bit quantization, etc.). In the long term, I personally look forward to more fundamental improvements to the mechanism itself emerging as the academic community achieves deeper understanding of the Attention mechanism.

But regardless, it is a "microcosm" of the success of deep learning thought as a whole — and this is why we chose to approach this work from the angle of Attention.

---

About the Interviewee

Dr. Zhao Tianchen, PhD candidate at the Energy-Efficient Computing Lab, Department of Electronic Engineering, Tsinghua University.

This interview was completed during his research tenure with ByteDance Seed-Vision team. His research focuses on efficient algorithm design for visual generation models and algorithm-hardware co-optimization. His work on sparsity, low-bit quantization, model construction, and corresponding system-level joint optimization for visual generation scenarios encompasses Attention mechanism optimization, with research results published at ICLR, ECCV, CVPR, DAC, and other conferences.

Paper: https://arxiv.org/abs/2506.16054

Our thanks to Dr. Tianchen for participating in this conversation.

Oasis Capital continues to warmly invite frontline researchers, developers, and thinkers worldwide to jointly advance this exploration of Attention.

If you're working on Attention-related research — whether in algorithms, systems, cognition, neuroscience, or product design and content construction — and would like to explore and discuss these topics with us, please reach out via the QR code at the end of this article!