Morning Star | Infinigence AI's Yu Wang: Token Has Become One of the Most Core Production Factors in the Intelligence Era

The "Qiming Venture Partners · Entrepreneurship and Investment Forum — Venture Capital Igniting the Resonance Cycle of AI Technology and Applications", hosted by Qiming Venture Partners at the 2025 World Artificial Intelligence Conference (WAIC), was successfully held on July 28 at the Blue Hall of Shanghai World Expo Center.

At this forum, Yu Wang, tenured professor and department head of the Department of Electronic Engineering at Tsinghua University, and initiator of Infinigence AI, delivered a speech titled "Advancing AI Infrastructure Evolution Through Hardware-Software Co-Design."

Yu Wang, tenured professor and department head of the Department of Electronic Engineering at Tsinghua University, and initiator of Infinigence AI

Wang pointed out that in the process of transforming artificial intelligence into actual productive force, Token — the fundamental unit of large model input and output — has become one of the most critical production factors in the intelligent era. The past value chain was driven by electrical energy improving computing power to accomplish simple tasks, but now it has evolved into converting electrical energy into computing power, then using that computing power to generate Tokens, ultimately supporting the execution of complex tasks. Along with this transformation, the core metric for evaluating infrastructure efficiency is also changing — the traditional "computing operations per joule of energy (TOPS/J)" is being replaced by "effective Tokens processed per joule of energy (Tokens/J)." How to optimize Token efficiency per unit of energy consumption will be the central proposition for infrastructure and system design in the AI 2.0 era.

Below are selected excerpts from Yu Wang's speech.

Thank you very much to Qiming Venture Partners for the invitation. I originally started from the AI infrastructure or AI hardware domain, and gradually moved into system-level work, so today I mainly want to share with you our practices over the past period, as well as some predictions about the future. We can't make refined predictions like AI can; we'll mainly discuss this from a technical perspective.

We deeply recognize that true AI can indeed form productive force. In the process of productivity enhancement, there is a very important dimension called "data," and we have been thinking: for AI infrastructure, what does it consume? Token is a very important unit. In the era of intelligent revolution, if the inputs are computing power and data, and what mainly powers them is electricity, then what is ultimately being processed? We now see a universal fundamental unit — Token. Token, as the input and output of artificial intelligence models, is actually crucial. Although at the macro level, the current production factor is called the data factor, in the large model era, the core production factor may be Token.

In the AI 1.0 era, each different vertical domain of data could train a vertical model, but now people convert all the world's data into Tokens, use these Tokens to train a Next Token Prediction large model, then output them as Tokens, and then these Tokens are converted into different modalities — only then can so-called multimodal applications be supported. Therefore, Token input and output become very critical. The corresponding paradigm also changes: in the past, it was from electrical energy to computing power to complete simple tasks; now it has transformed into from electrical energy to computing power, then to processing Tokens, and finally completing complex tasks.

The perspective for evaluating infrastructure also changes accordingly: previously, the focus was on computing operations output per second or per joule; now it has become the number of Tokens supported per joule or per watt. So in the AI 2.0 era, what needs to be considered is the change in Tokens under unit power consumption or unit energy consumption.

In the AI 1.0 era, all chip evaluation standards were based on TOPS per joule, or TOPS per watt; now, we believe that evaluation metrics need to change from AI 1.0 to 2.0, because those in academia always want to use a unified standard to evaluate and thereby guide the development of the entire system, and Token is precisely such a suitable standard.

As artificial intelligence or AGI develops through different stages, the definition of the problem also evolves: under the premise of ensuring that intelligence levels exceed a certain threshold, optimizing Tokens/J as much as possible serves as an important guiding metric for the next step in infrastructure and system design dimensions.

For example, as just mentioned, energy efficiency demands have several different intelligence levels, from the initial intelligent conversational assistant, to logical Q&A, to Embodied Artificial Intelligence control. Currently, we see some work has already reached 10 Tokens/J. When intelligence requirements become higher, the energy efficiency demands for Tokens/J actually differ greatly — at level 2, the difference is 1-2 orders of magnitude, and at level 3, it's 2-3 orders of magnitude. Filling these gaps requires technical efforts, solving this problem through software and hardware collaborative optimization.

From simple dialogue, to Olympiad math problem solving, interactive scenarios, to AI applications in scientific research (AI for Science), and even to the emergence of swarm intelligence — as capabilities improve level by level, we need to ensure the continuous evolution of "high-quality Tokens/J" without compromising the improvement of large model inference performance.

There is a book called Thinking, Fast and Slow. In it, the human cognitive system is divided into two categories: System 1 is responsible for intuition and perception — given an input, there is immediately an output. The other system is System 2, which is more like a slow thinking process. So when large models first emerged, I was thinking whether this was System 2? After some consideration, I concluded it wasn't. It was only when OpenAI's o1 came out that it seemed to approach System 2, because it has reasoning based on chain-of-thought. So we see that the second type of system is actually an action-by-action reasoning process.

First, let's look at the first type of system, which we have studied for a long time. Starting from image recognition, face recognition — given an image, output a result. At this point, if we want to improve Tokens/J, what can we do? We know that whether in the past or present, large models are essentially neural networks, and the core of neural networks is matrix operations. Total computational cost equals computation amount × single operation cost. Here, computation amount refers to the number of non-zero elements in the matrix — only non-zero elements need multiplication and addition operations. Another factor is how long a vector is needed to store the description of each non-zero element. These were two very important fundamental optimization directions we identified at that time.

Why can large models/neural networks be optimized? The traditional model is to do algorithm design first, then hardware design afterward. But because neural networks have learnable characteristics, these two stages can be optimized together, thereby ensuring algorithm accuracy remains unchanged while making more elements in the matrix become 0, or making elements in the matrix become shorter vectors (bits). This is the work we have been doing for a long time. The number of neuron connections in the human brain doesn't seem large — each neuron is connected to less than 10% of other neurons, meaning that even with billions of parameters, the actual effective connections are only on the order of less than 100 million. So we hope to learn from biological development to make model matrices sparse. Of course, in exploring sparsification, we later saw three trends:

First, the sparsification target has developed from weight sparsity to activation sparsity. This is because in the large model era, the main bottleneck of models has shifted to activation value computation in the attention mechanism, making activation sparsity the research focus. Second, sparsity patterns have developed from unstructured to structured sparsity, because hardware itself is densely structured — it can only compute well with structured sparse data; the non-zero elements can't appear randomly, but must be regular non-zero elements. Third, the trend has developed from post-training sparsity to sparse training. DeepSeek is also making attempts in this area — making matrices sparse during training, so that the proportion of zero elements can further increase from 50% to 75%.

Another important direction is quantization. Because the human brain operates through pulses, and pulses are essentially 0 and 1, which is vastly different from the 32-bit floating point or 64-bit floating point commonly used in our real computing systems. One is 64-bit length, the other is simple 0/1 signals. So now in neural network optimization, it can be further reduced to 8-bit, 4-bit, etc. Looking back over the past 10-plus years, from 32-bit floating point to 8-bit fixed point — this was a consensus formed around 2015-2016. Then from INT8 to INT4 or binary. Of course, binary solutions are still being worked on in the industry, with not very good results yet. So what everyone sees now is basically FP8, FP4, which is also the data representation supported in NVIDIA chips.

Now domestic chips are also gradually supporting FP8, FP4, to support more efficient training and inference work. From the matrix dimension, which is most core to the first type of system, matrix operations are actually the foundation of model inference, and whether it's model training or large-scale Agent inference, the support of the entire computing system is needed. The effective computing power of the entire system depends on the total number of cards in the cluster and the computing power of single chips. Specifically, the current technical path starts from on-wafer level integration, integrating multiple chips on the same silicon substrate. If a silicon substrate hosts an independent chip, then the next step is to achieve inter-chip interconnection within a single machine, and then further achieve interconnection between servers. By combining multiple supernodes, it ultimately becomes an overall server cluster. Therefore, building a cluster with high total computing power is a process of step-by-step integration and interconnection from underlying chips to upper-level architecture. Every link in this process — from chip design, interconnection technology to cluster architecture — has significant optimization space, which also provides various possibilities for overall computing power improvement.

We have been talking about doing optimization for "small box," "medium box," and "large box" — each level has different focuses. At the small box level, optimization mainly concentrates on computation itself; at the medium box level, the focus shifts to connection and coordination between different chips, including optimization of computing pipelines and multi-dimensional parallel strategies; at the large box (thousand-card, ten-thousand-card, and even hundred-thousand-card) dimension, what matters more is how to achieve good resource scheduling, and how to ensure system stability — for example, at ten-thousand-card or even hundred-thousand-card scale, how to ensure correctness of task execution, whether rapid detection and recovery are possible when faults occur, and how to achieve effective fault tolerance management. Therefore, in terms of system optimization, there are still many problems to be solved, and everyone in the industry is working very hard to continuously advance related work.

On the device side, there is still roughly a 10x gap between our application needs and actual capabilities. Of course, we saw at this year's WAIC that a peak 200 Tokens/s edge solution has already been proposed. We very much look forward to seeing these breakthroughs truly meet people's daily life needs in practical use as soon as possible. On the cloud side, the industry is also actively advancing optimization from the energy efficiency perspective, improving per-node Tokens/J energy efficiency, thereby achieving more efficient and sustainable computing.

From the infrastructure layer perspective, on one hand, we need to optimize ecosystem differences and achieve optimization of the current heterogeneous multi-vendor ecosystem status; on the other hand, we need to do joint optimization at different levels, from the underlying infrastructure layer, to the middle model layer, to the upper application layer, in order to build better AI infrastructure design.

Here, we also showcase what Infinigence AI has achieved and supports — from underlying infrastructure to upper-level AI applications, multi-vendor computing power scheduling and optimization work. Currently, our clusters can provide services for major computing power consumers in multiple provinces and cities across China, including Beijing, Shanghai, etc. We hope that the next step can be to further improve the efficiency of computing power application on the basis of these public platforms.

In the process of practical application, many different problems will also arise. What we discussed just now was training or inference of a single model. But when downstream tasks develop from a single large model to intelligent agent collaboration, multiple models will appear simultaneously on one cluster, and this occurs during both single-task scheduling and multi-task scheduling, so GPU utilization will decrease, dynamics will become stronger, and overhead will be large — these are all problems we are currently solving.

Second, when developing from a single intelligent agent to multiple intelligent agents, many problems also arise. Including parallelism, context processing, tool complexity, etc., challenges will be faced at various different windows, capacities, and energy ceilings, so the system needs further tools and system optimization to support the development from single-agent to multi-agent.

Third, today's forum will later discuss Embodied Artificial Intelligence. Embodied Artificial Intelligence will push the research focus from cloud-side chips to edge-side inference evolution, marking that intelligent systems will move from the digital world to the physical world, truly having an effect on the physical world. If we want to achieve very good results on the device side — for example, deploying a 7B parameter-scale model — it basically needs to achieve over 20 Tokens/J in energy efficiency, while also achieving 100-200 Tokens/s inference speed. This is a very difficult technical challenge, and also a major problem faced in the process of intelligent systems moving from digital to physical world.

Finally, we hope that Infinigence AI can collaborate with upstream and downstream partners in the industry chain to jointly build a new generation of cloud and terminal devices. Currently, we are actively adapting to various edge-side chips, continuously serving numerous partners including Lenovo. In the future, we hope to both empower thousands of industries through AI cloud capabilities, and help new terminal devices enter millions of households.

Source | IPO Zaozhidao

Previous Highlights

Qiming Stars | Infinigence AI's Yu Wang: Exploring the Next Stop of Edge Intelligence Qiming Perspectives | Qiming Venture Partners' Alex Zhou: 2025 Will Be a Big Year for Comprehensive AI Application Deployment Qiming Honors | Qiming Venture Partners' Duane Kuang, William Hu, and Alex Zhou Named 2025 PEDaily TOP100 Investors

Founded in 2006, Qiming Venture Partners currently manages 11 USD funds and 7 RMB funds, with total assets under management reaching $9.5 billion. Since its establishment, the firm has focused on investing in outstanding early and growth-stage enterprises in Technology and Healthcare innovation.

To date, Qiming Venture Partners has invested in over 580 high-growth innovative companies, of which more than 210 have gone public on the New York Stock Exchange, NASDAQ, Hong Kong Exchanges and Clearing Limited, Shanghai Stock Exchange, and Shenzhen Stock Exchange, or exited through M&A and other means. Over 80 companies have become recognized unicorns or super-unicorns in their industries.

Many companies in Qiming Venture Partners' portfolio have grown into the most influential companies in their respective fields, including Xiaomi (01810.HK), Meituan (03690.HK), Bilibili (NASDAQ:BILI, 09626.HK), Zhihu (NYSE:ZH, 02390.HK), Roborock (688169.SH), UBTECH (09880.HK), WeRide (NASDAQ:WRD), Insta360 (688775.SH), Gan & Lee Pharmaceuticals (603087.SH), Tigermed (300347.SZ, 03347.HK), Zai Lab (NASDAQ:ZLAB, 09688.HK), CanSino Biologics (688185.SH, 06185.HK), Schrödinger (NASDAQ:SDGR), MicroPort EP MedTech (688617.SH), Sanyou Medical (688085.SH), Amoy Diagnostics (300685.SZ), Berry Genomics (000710.SZ), SinoCellTech (688520.SH), Yuanxin Technology, Insilico Medicine, MediLink Therapeutics, LaNova Medicines, Zhipu AI, StepFun, Biren Technology, and others.